What Is a robots.txt File?
A robots.txt file tells search engine crawlers which pages they can and cannot access on your website. It lives at yourdomain.com/robots.txt and is one of the first files Googlebot requests when crawling a site.
Our free robots.txt generator helps you create a properly formatted file with user-agent rules, allow/disallow directives, sitemap references, and optional crawl-delay settings — without writing syntax by hand.