robots.txt is a small plain-text file at a site's root that tells well-behaved crawlers which parts of the site they're welcome to fetch. It's a request, not a lock — it doesn't hide or protect anything, it just asks compliant bots (Google, Bing, and similar) to stay out of the paths you list.
Steps
- Open Robots.txt Generator.
- Pick a preset — Allow all, Block all, or Custom (list specific paths to disallow).
- Optionally add your sitemap URL, then download
robots.txtand place it at your site's root, exactly atyoursite.com/robots.txt.
The three rules that matter most
User-agent names which crawler a block of rules applies to — * means every crawler that doesn't have its own specific block. Disallow lists a path pattern to keep out (an empty Disallow: means nothing is blocked). Allow explicitly permits a path, mainly useful for carving out an exception inside an otherwise-disallowed folder. A Sitemap: line pointing to your sitemap's full URL is optional but genuinely useful — it's one of the few places a crawler reliably looks for it.
What robots.txt doesn't do
- It doesn't remove an already-indexed page from search results — a page can still appear in Google if it's linked from elsewhere, even if crawling is disallowed. Use a
noindexmeta tag for that instead. - It doesn't stop non-compliant scrapers — robots.txt is an honor system; a bot that ignores it faces no technical barrier here.
- It doesn't need a
Disallowline for pages you want crawled — the default, with no file present at all, is that everything is allowed anyway.