A robots.txt file is a plain text file at the root of your site that tells search engine crawlers which parts of the site they may request and which to skip. The generator below builds a valid robots.txt for you as you type, entirely in your browser, so you can copy it, save it as robots.txt, and upload it to your domain root.
What robots.txt Is and What It Controls
Robots.txt is a text file that lives at the root of a domain and follows the Robots Exclusion Protocol. When a crawler such as Googlebot or Bingbot visits a site, it requests this file first and reads the rules before fetching other pages. The file controls crawling, which is the act of a bot requesting your pages. It groups rules under a User-agent line that names the crawler, then lists Disallow rules for paths the crawler should not request and Allow rules for paths it may request. A crawler that respects the protocol obeys these rules; the file does not enforce anything on bots that ignore it.
How to Use This Generator
- Enter the User-agent the rules apply to. An asterisk means every crawler, which is the common choice.
- Pick the default policy: allow all to let crawlers reach the whole site, or disallow all to block the whole site.
- Add the folders you want kept out of crawling, one path per line, such as /wp-admin/ or /cart/.
- Paste your sitemap URL so crawlers can find your full list of pages.
- Copy the file the tool builds, save it as robots.txt, and upload it to the root of your domain.
The robots.txt Directives
| Directive | What it does |
|---|---|
| User-agent | Names the crawler the rules below apply to. An asterisk matches all crawlers. |
| Disallow | Tells the named crawler not to request paths that start with the given value. A single slash blocks the whole site. |
| Allow | Tells the crawler it may request a path, used to carve an exception out of a broader Disallow. |
| Sitemap | Points crawlers to the full URL of your XML sitemap so they can discover your pages. |
Where to Place robots.txt
The file must sit at the root of the domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there, so a robots.txt in a subfolder is ignored. Each subdomain needs its own file, because the rules for blog.example.com are separate from those for www.example.com. The file applies to the protocol and host it is served from, so an HTTPS site reads the file served over HTTPS.
Common Mistakes
The most damaging mistake is a stray Disallow: / left over from a staging site, which blocks the entire site from crawling. Another is using robots.txt to hide a page and expecting it to vanish from search; blocked pages can still be indexed by URL. Blocking CSS or JavaScript files prevents search engines from rendering the page as a visitor sees it, which can hurt how the page is understood. Path matching is case sensitive and matches by prefix, so /Admin and /admin are different, and Disallow: /page blocks every URL that begins with /page. Place the file at the root, not in a folder, or it will not be read.
When to Use It
Use robots.txt to keep crawlers out of areas that waste crawl budget or add no value to search, such as admin folders, internal search result pages, cart and checkout paths, and faceted filter URLs that generate near-duplicate pages. Use it to point crawlers to your sitemap. Do not use it to protect private data or to remove a page from the index; those jobs belong to authentication and the noindex tag.
Last Thoughts on Generating robots.txt
A robots.txt file is a short set of instructions that shapes how crawlers spend their time on your site. The risk is not complexity but a single wrong line, so build it carefully, keep it small, and test it before you rely on it. Allow the pages you want found, block the folders that add no search value, and always point to your sitemap.
Generate your file above, then save it as robots.txt at your domain root. For related work, build your page tags with the meta tag generator, keep titles and descriptions in range with the meta length checker, and explore the rest of our free online tools.
Key Takeaways:
- Robots.txt controls crawling, not access; it asks well-behaved crawlers which paths to skip.
- The file must live at the domain root and be reachable at /robots.txt, with a separate file per subdomain.
- A Disallow rule does not keep a page out of the index; use a noindex tag or authentication for that.
- Group rules under a User-agent line, then add Disallow and Allow paths and a Sitemap line.
- Avoid a leftover Disallow: / and do not block CSS or JavaScript that crawlers need to render the page.
- This generator builds the file in your browser; nothing is sent anywhere.
Frequently Asked Questions (FAQs)
