robots.txt
A root-level text file that tells crawlers which paths they may or may not request.
robots.txt is a plain-text file at the root of a domain (/robots.txt) that tells crawlers which paths they are allowed to request. It uses User-agent, Allow, and Disallow directives, and can also point to your sitemap with a Sitemap: line.
It controls crawling, not indexing. Disallowing a URL stops compliant crawlers from fetching it, but a blocked URL can still appear in search results if other pages link to it — Google just cannot see its content. To keep a page out of the index, use a noindex tag on a crawlable page, not a Disallow rule.
A frequent mistake is blocking resources like CSS or JavaScript, which can break how Google renders and understands a page. Keep robots.txt narrow, and include a Sitemap: reference so crawlers can find your sitemap without a manual submission.
Related terms
Stop worrying about your sitemap.
Indexly crawls your site, builds the sitemap, hosts it, and keeps it current — automatically.
Get started free