Files stay on your device Use Robots.txt Generator ↓

Robots.txt Generator Online — Free & Private

Write a simple robots.txt that allows crawling and points at a sitemap.

No sign-up required Files stay on your device Works offline once loaded

Loading the tool workspace…

Create a robots.txt file for an HTTP or HTTPS site origin. Leave the optional disallowed-path list empty to allow crawling, or enter one path per line to request that compliant crawlers avoid it. This controls crawling, not authentication or whether a linked URL can appear in search.

The file is generated in the browser from the URL you type. It is not uploaded.

Large files and OCR need sufficient device memory; review the downloaded result before using it.

Write a simple robots.txt that allows crawling and points at a sitemap. Everything runs in this tab — nothing is uploaded for the core job.

  1. Enter your site origin, for example https://example.com.
  2. Optionally add disallowed paths, one per line, beginning with /.
  3. Generate and review every rule.
  4. Save robots.txt at the origin root and test the deployed file.

Privacy proof for Robots.txt Generator

This tool processes input in your browser. Open DevTools → Network while you run it to check the requests. Some tools need downloaded libraries on their first use.

Popular available tools for everyday tasks. Browse categories or search the full catalog.

Star any tool — it appears under Favorites on this device.

Available tools process your files and text on your device. Choose the operation you need, check its supported inputs and limits, and keep the original file until you have reviewed the result. PDF compression offers a lossless mode and optional image-based modes; image-based exports lose selectable text. Scanned PDFs need OCR for text extraction. OCR language models may need an initial download. Some catalog operations are still being completed and are clearly marked as unavailable. A current browser and sufficient device memory are required. Optional sign-in stores activity metadata, not your documents.

Can I disallow /admin/?

Yes. Enter /admin/ in the optional path list. It does not protect access; use authentication for private data.

What if I enter a page URL?

Use only the origin. Page paths and query strings are rejected.

Will this stop all scraping?

No. Crawlers can ignore these rules.

Does this remove a page from search?

Not reliably. Use appropriate indexing directives on crawlable pages and access controls for private information.

From our blog: What Is robots.txt, and How Do You Write One?