Sitemap vs Robots.txt: What Each Does (And Why You Need Both)

The short answer to "sitemap vs robots.txt" is that they do opposite jobs: your sitemap tells search engines which pages exist, and your robots.txt tells them which pages not to crawl. You need both, they're not substitutes, and confusing them is the source of a surprising number of indexing problems.

This guide clears up what each one actually does, where they overlap, where they don't, and the handful of mistakes that trip up almost everyone the first time they edit either file.

Table of contents

1. The one-sentence difference
2. What robots.txt actually does
3. What an XML sitemap actually does
4. Side-by-side comparison
5. Do I really need both?
6. How they work together (the correct setup)
7. Common mistakes
8. FAQ

---

The one-sentence difference

Robots.txt controls crawling. A sitemap describes what exists.

Everything else is commentary. If you remember only that, you'll get 90% of the decisions right.

Or, with slightly more nuance:

• A sitemap is a list of URLs you want search engines to know about.
• A robots.txt is a list of rules about which URLs search engines are allowed to fetch.

One is inclusive ("here's what I have"). The other is exclusive ("here's what not to touch"). Together they give Google a clean picture of your site: what's there, and what's not their business.

What robots.txt actually does

robots.txt is a plain text file at the root of your site — https://example.com/robots.txt — that tells crawlers which parts of your site they're allowed to fetch.

Here's a minimal, correct example:

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

That says: every bot is welcome everywhere, and by the way, here's my sitemap.

A more realistic one for a typical site:

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /api/
Disallow: /?utm_source=
Sitemap: https://example.com/sitemap.xml

The rules in plain English:

• User-agent: * — these rules apply to all bots.
• Disallow: /admin/ — don't crawl anything under /admin/.
• Disallow: /*?utm_source=* — don't crawl URLs with a utm_source query parameter.
• Sitemap: — here's where my sitemap lives.

What robots.txt can do

• Block crawlers from specific paths (Disallow:)
• Explicitly allow exceptions within blocked sections (Allow:)
• Point to your sitemap (Sitemap:)
• Apply different rules to different bots (User-agent: Googlebot vs User-agent: Bingbot)

What robots.txt cannot do

This is where people get hurt:

• It does not prevent indexing. Blocking a URL in robots.txt stops Google from crawling it, but if the URL is linked from elsewhere, Google can still index it — just without any content. You'll see it show up in search results as a naked URL with "No information is available for this page" underneath.
• It does not hide pages. robots.txt is publicly readable. Anyone can visit yoursite.com/robots.txt and see every path you tried to hide. Do not use it as a security mechanism.
• It does not remove indexed pages. If a page is already in Google's index, adding it to robots.txt does nothing to remove it — and actually prevents Google from seeing the noindex tag that would remove it.
• It no longer supports noindex: directives. Google dropped support for Noindex: lines inside robots.txt on September 1, 2019 — part of retiring unsupported rules in the robots exclusion protocol. Anything you see in old tutorials telling you to write Noindex: /path/ in robots.txt is out of date. Use a <meta name="robots" content="noindex"> tag on the page instead.

What an XML sitemap actually does

An XML sitemap is a file listing every URL on your site you want search engines to know about, in a specific format:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-04-15</lastmod>
  </url>
  <url>
    <loc>https://example.com/blog/intro-to-seo</loc>
    <lastmod>2026-04-12</lastmod>
  </url>
</urlset>

What a sitemap can do

• Tell Google the full list of URLs on your site in one place
• Indicate when each URL last changed (via <lastmod>)
• Help Google discover pages with weak or no internal linking
• Surface indexing problems in Search Console's coverage reports
• Speed up discovery for new content on established sites

What a sitemap cannot do

• It does not force indexing. Google decides whether each URL is worth keeping. A sitemap entry is a suggestion, not a command.
• It does not block crawling. There's no "don't crawl this" in a sitemap — that's robots.txt's job.
• It does not override quality judgments. Submitting a thin page in a sitemap doesn't convince Google to index it. We cover this in why Google isn't indexing your pages.
• It doesn't affect rankings directly. Being listed in a sitemap doesn't make a page rank higher. It just helps the page get found.

For a full walkthrough of sitemap format, limits, and best practices, see our XML sitemap guide.

Side-by-side comparison

	Sitemap	Robots.txt
Purpose	Lists URLs you want crawled	Tells bots what they can/can't crawl
Format	XML	Plain text
Location	Anywhere (typically /sitemap.xml)	Must be at root (/robots.txt)
Required?	Strongly recommended	Strongly recommended
Controls crawling?	No	Yes
Controls indexing?	No	No (not directly)
Can hide pages?	No	No (publicly readable)
Max size	50MB / 50,000 URLs per file	500 KiB (Google)
How Google finds it	Via GSC submission or robots.txt link	Automatic — always fetched from /robots.txt
Effect on a new page	Gets discovered faster	Gets crawled (or not)
Effect on a junk page	If listed, Google notices junk	If disallowed, Google doesn't fetch it

Do I really need both?

Yes. They solve different problems and neither is optional for a site that takes SEO seriously.

Why you need robots.txt

Even if you don't want to block anything, robots.txt is where you tell crawlers where your sitemap lives. Without it, Google still has to discover your sitemap some other way (usually through Search Console submission).

More importantly, robots.txt saves your crawl budget. Google allocates a finite amount of crawling per site per day. If Googlebot is wasting it on your admin dashboard, internal search pages, filtered product URLs, or tracking-parameter variants, that's crawl budget stolen from pages that matter. We cover this in detail in crawl budget explained.

Why you need a sitemap

Internal linking alone rarely covers your whole site. Every site has:

• Deep pages more than 3–4 clicks from the homepage
• Orphan pages with no internal links
• Recently published pages not yet linked from anywhere
• URLs that appear only through dynamic navigation

A sitemap catches all of them. For sites over a few hundred pages, and especially for e-commerce, SaaS, and publishers, a sitemap is the difference between Google knowing about 95% of your pages and Google knowing about 60%.

Neither replaces the other

Robots.txt without a sitemap: Google finds your pages slowly, if at all, and misses anything not linked from the homepage.

Sitemap without robots.txt: Google crawls everything, including the junk, and wastes crawl budget on pages you never wanted indexed.

How they work together (the correct setup)

The clean pattern that works for 95% of sites:

Step 1: Write a minimal robots.txt

User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /checkout/
Disallow: /*?sessionid=
Disallow: /search?
Sitemap: https://example.com/sitemap.xml

Disallow paths that genuinely shouldn't be crawled: admin areas, user accounts, cart/checkout, internal search results, URLs with tracking parameters. Leave everything else open.

Step 2: Generate a sitemap that lists only indexable pages

Your sitemap should contain:

• Every page you want indexed
• Only canonical URLs (no duplicates, no parameter variants)
• No noindex pages
• No 404s or redirects
• No URLs blocked by your robots.txt

That last one catches people off guard. Here's why it matters.

The silent conflict: disallowed pages in a sitemap

If a URL is blocked in robots.txt but listed in your sitemap, Google gets contradictory instructions: "here's a page I want you to know about" + "you can't fetch it." Google will flag this in Search Console as "Indexed, though blocked by robots.txt" — meaning it might index the URL as a naked link with no content. Not what you want.

The fix is simple: your sitemap and robots.txt should never disagree about a URL. If robots.txt disallows /admin/, /admin/* should not be in your sitemap. Keeping the two in sync manually is tedious. Tools like Indexly handle this automatically — the crawler respects your robots.txt by default, so disallowed URLs never make it into the generated sitemap.

Step 3: Submit the sitemap in Google Search Console

Two places Google can find your sitemap:

1. The Sitemap: line in your robots.txt (automatic)
2. Search Console → Sitemaps → Add a new sitemap (manual)

Do both. The Search Console submission is what unlocks the coverage and indexing reports you'll actually use to debug indexing problems. Full walkthrough in submitting your sitemap to Google Search Console.

Step 4: Keep the sitemap fresh

A sitemap that's six months out of date is worse than no sitemap — Google starts trusting it less. The fix is automation: have something crawl your site on a schedule and regenerate the sitemap whenever pages are added or removed. Indexly does this as its core job; so do most CMS sitemap plugins within a single platform.

Common mistakes

The mistakes we see most often, in rough order of frequency:

1. Using Disallow: to deindex a page. Blocking a URL in robots.txt doesn't remove it from the index. If anything, it prevents Google from seeing the noindex tag that would. Use <meta name="robots" content="noindex"> on the page itself, and let Google crawl it one last time.

2. Writing Noindex: in robots.txt. Unsupported since 2019. Silently ignored by Google. If your old robots.txt has these, delete them and add proper noindex meta tags on the affected pages.

3. Disallow: / left in production from staging. Blocks Google from everything. Classic. Check your robots.txt after every deployment, especially after a major infrastructure change.

4. Listing noindex pages in the sitemap. The sitemap says "index this", the page says "don't index this." Google picks the page's instruction and flags the sitemap as unreliable. Only indexable pages go in the sitemap.

5. Listing URLs in the sitemap that are blocked by robots.txt. Same contradictory-signal problem. Audit regularly.

6. Treating robots.txt as access control. It's a suggestion, not a wall. Malicious bots ignore it, and the file is publicly readable. Anything you genuinely need to protect needs server-side auth — not a Disallow: line.

7. Putting the sitemap at the wrong URL and forgetting to update robots.txt. If your sitemap lives at /sitemaps/main.xml but robots.txt still points at /sitemap.xml, Google fetches a 404 and logs it as a broken sitemap.

8. Blocking CSS or JavaScript in robots.txt. Google needs to fetch your CSS and JS to render the page. Blocking them makes Google see a broken, unstyled version — which gets scored as poor UX. A common 2010s-era habit that still shows up in robots.txt files today.

---

FAQ

What's the difference between a sitemap and robots.txt?

A sitemap is an inclusive list: "here are the URLs on my site you should know about." Robots.txt is an exclusion list: "here are the paths bots should not crawl." They complement each other — the sitemap helps discovery, robots.txt controls what gets fetched. You need both, and they should not disagree about any URL.

Can I use robots.txt to stop a page from appearing in Google?

No. Robots.txt blocks crawling, not indexing. If a URL is blocked in robots.txt but linked from somewhere, Google can still index it as a bare URL with no content. To remove a page from the index, use a <meta name="robots" content="noindex"> tag on the page itself and make sure it's crawlable.

Should the sitemap be listed in robots.txt?

Yes. Adding Sitemap: https://yoursite.com/sitemap.xml to your robots.txt is the standard way to announce your sitemap to any crawler that reads the file. It's not a replacement for submitting the sitemap in Google Search Console, but it's a free signal and takes one line.

What happens if a URL is in my sitemap but blocked by robots.txt?

Google treats it as a contradictory signal. You'll typically see "Indexed, though blocked by robots.txt" in Search Console, meaning Google may index the URL as a naked link with no content. Fix by removing the URL from either the sitemap or the robots.txt Disallow rule — whichever reflects your actual intent.

Do small sites need a robots.txt?

Technically no — without one, crawlers assume everything is allowed. Practically yes, because it's the standard place to announce your sitemap, and you'll almost certainly want to disallow some paths eventually (admin, cart, internal search). Even a three-line robots.txt is worth having from day one.

Is robots.txt case-sensitive?

The paths are, yes. Disallow: /Admin/ does not block /admin/. Directive names (User-agent, Disallow, Allow, Sitemap) are case-insensitive. Match the exact casing of the URLs on your site — if your admin panel lives at /Admin/, that's what goes in the rule.

---

The bottom line

Robots.txt and sitemap aren't competing tools — they're the two halves of how you describe your site to Google. Robots.txt controls what gets crawled. The sitemap describes what exists. Between them, you should be able to answer "which URLs on my site should Google know about, and which should it leave alone?" clearly and consistently.

The catch is keeping them in sync as your site grows. Every new section, every deprecated URL, every parameter experiment is a chance for the two files to drift apart. If you'd rather not babysit that, Indexly keeps your sitemap current automatically and respects your robots.txt by default — so the two never contradict each other. Free plan, no credit card, one minute to set up. Every page found. Every page indexed.