Robots.txt Generator Online — Free & Private
Write a simple robots.txt that allows crawling and points at a sitemap.
Robots.txt Generator
Loading the tool workspace…
Create a robots.txt file for an HTTP or HTTPS site origin. Leave the optional disallowed-path list empty to allow crawling, or enter one path per line to request that compliant crawlers avoid it. This controls crawling, not authentication or whether a linked URL can appear in search.
The file is generated in the browser from the URL you type. It is not uploaded.
Large files and OCR need sufficient device memory; review the downloaded result before using it.
- Avoids a fake “block all AI” list that goes stale.
- Puts the sitemap line in the place crawlers look.
- Does not pretend robots.txt is access control.
- Local only.
How to use Robots.txt Generator
Write a simple robots.txt that allows crawling and points at a sitemap. Everything runs in this tab — nothing is uploaded for the core job.
- Enter your site origin, for example https://example.com.
- Optionally add disallowed paths, one per line, beginning with /.
- Generate and review every rule.
- Save robots.txt at the origin root and test the deployed file.
Privacy proof for Robots.txt Generator
This tool processes input in your browser. Open DevTools → Network while you run it to check the requests. Some tools need downloaded libraries on their first use.
Popular tools
Popular available tools for everyday tasks. Browse categories or search the full catalog.
Star any tool — it appears under Favorites on this device.
Merge PDF
Combine several PDFs into one, in the order you choose.
Split PDF
Cut one PDF into two at a page you pick.
Compress PDF
Shrink file size by re-encoding each page.
PDF to JPG
Export every page as a high-res image.
JPG/PNG to PDF
Turn a stack of photos into one PDF.
Watermark PDF
Stamp a text or image watermark — positioned, tiled, or layered behind content.
Sign PDF
Draw a signature and place it on any page.
OCR PDF
Recognize text in scanned or photographed PDFs, then get a searchable PDF or a plain text file.
Image Compressor
Compress JPG without losing quality — shrink images for the web without uploading.
JSON Formatter
Format, validate and minify JSON, with clear error messages.
EMI Calculator
Calculate the monthly EMI for a loan amount, rate and tenure.
QR Code Generator
Turn text or a URL into a downloadable QR code image.
How this actually works
Available tools process your files and text on your device. Choose the operation you need, check its supported inputs and limits, and keep the original file until you have reviewed the result. PDF compression offers a lossless mode and optional image-based modes; image-based exports lose selectable text. Scanned PDFs need OCR for text extraction. OCR language models may need an initial download. Some catalog operations are still being completed and are clearly marked as unavailable. A current browser and sufficient device memory are required. Optional sign-in stores activity metadata, not your documents.
Questions people actually ask
Can I disallow /admin/?
Yes. Enter /admin/ in the optional path list. It does not protect access; use authentication for private data.
What if I enter a page URL?
Use only the origin. Page paths and query strings are rejected.
Will this stop all scraping?
No. Crawlers can ignore these rules.
Does this remove a page from search?
Not reliably. Use appropriate indexing directives on crawlable pages and access controls for private information.
From our blog: What Is robots.txt, and How Do You Write One?