Robots.txt Generator
Generate clean, SEO-compliant robots.txt files to manage search engine crawl permissions, direct sitemaps, and block unwanted bots.
Standard Directives
Custom Paths & Exceptions
Generated robots.txt
# Created using Robots.txt Generator by UnifiedTools Pro (https://unifiedtoolspro.xyz) User-agent: * Disallow: /wp-admin/ Disallow: /tmp/ Disallow: /private/
What is robots.txt?
Robots.txt is a text file that resides in your website's root directory. It informs web crawlers (like Googlebot) which pages or folders they can or cannot request from your server.
Note: Robots.txt rules do not guarantee page indexing prevention (use a `noindex` meta tag for that). It primarily regulates search crawl traffic and bandwidth consumption.