Free Robots.txt Generator

Build a correct robots.txt without memorizing crawler syntax. Add rules per user-agent — allow and disallow paths with a few clicks — attach your sitemap, set a crawl delay, and copy a clean file ready to upload to your domain root.

Crawler rules

Generated robots.txt


    

    



  
Private by design. This tool runs 100% in your browser — your code is never uploaded to any server.

  

    
How to Use the Robots.txt Generator

    
    
      
  1. Add a rule row for each crawler (or * for all).
    2. 
      
  2. Set each row to Allow or Disallow and enter the path.
    3. 
      
  3. Add your sitemap URL and an optional crawl delay.
    4. 
      
  4. Copy or download robots.txt and upload it to your domain root.
    5. 
    

  

  

    
Features

    
Rule rows pair a user-agent field with allow/disallow toggles and path inputs, supporting unlimited rows for granular crawler control. The builder appends your XML sitemap URL, an optional crawl-delay directive, and comments explaining each section — output follows the robots exclusion protocol exactly.

    
Benefits & Use Cases

    
Site owners keep staging areas, admin panels, and faceted-search URLs out of search indexes; SEOs point crawlers at sitemaps for faster discovery; developers scaffold the file for new projects in a minute. A correct robots.txt prevents wasted crawl budget on junk URLs.

    
Tips

    
Remember robots.txt is advisory — well-behaved crawlers obey it, malicious bots ignore it. Never rely on robots.txt to hide truly sensitive content; use authentication instead. After uploading, test with Google Search Console’s robots.txt tester.

  

  

    
Frequently Asked Questions

    
What is robots.txt?
A plain-text file at your domain root (example.com/robots.txt) that tells search-engine crawlers which URLs they may or may not request. It guides crawling behavior across your site.

    
What is the difference between Allow and Disallow?
Disallow blocks crawlers from the given path prefix; Allow explicitly permits a path, which is useful for punching exceptions inside a broader Disallow rule.

    
Does robots.txt improve my rankings?
Not directly. It helps by keeping crawlers focused on important pages (saving crawl budget) and out of duplicate or private areas — which indirectly supports SEO health.

    
Can robots.txt keep a page out of Google?
It stops crawling, but Google may still index a URL it discovers via links (showing it without a description). For true exclusion, use noindex or authentication.

    
Where do I put the file?
At the root of your domain: https://example.com/robots.txt. Crawlers only check that exact location.

  

  

    
Related Tools

    

      
Sitemap Generator
Generate XML sitemaps free with NToolHub. Paste URLs, set priority and change frequency — valid sitemap.xml in seconds, 100% private.
Open tool →
      
.htaccess Generator
Build Apache .htaccess rules visually with NToolHub’s free generator. HTTPS, www, redirects, hotlink protection — copy-ready server config.
Open tool →
      
Meta Tag Generator
Generate SEO and social meta tags with NToolHub’s free tool. Title, description, Open Graph, Twitter cards — copy-ready HTML head code.
Open tool →
      
URL Encode
Percent-encode URLs and text instantly with NToolHub’s free tool. Component or full-URL mode, UTF-8 safe — private in-browser.
Open tool →