Robots.txt Generator

Generate a robots.txt file for a public site, a blocked staging site, or a common WordPress-style setup.

Preset

Site URL

User-Agent

Crawl Delay

Custom Disallow Paths

Custom Allow Paths

Include sitemap line

Summary

User Agents

Disallow Rules

Allow Rules

Sitemaps

Generated robots.txt

Upload this plain-text file to your site root.

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

Related Tools

Check robots.txt file

Parse XML sitemaps and sitemap index files

Extract meta tags from HTML

Find favicon for a website

Check for broken links on a page

Check if images are optimized