Crawling

Crawling is how search engines fetch pages from the web. A crawler like Googlebot starts from known URLs, requests each one, reads the HTML, and follows the links it finds to discover more pages. It also reads sitemaps and robots.txt to decide what to fetch.

Crawling comes before indexing: a page must be fetched before it can be considered for the index. Pages that are hard to reach — buried deep in the site, reachable only through forms, or not linked at all — may be crawled late or missed. Sitemaps help here by giving crawlers a direct list of URLs.

Crawling respects robots.txt disallow rules and server response codes. Slow servers, frequent errors, or heavy redirect chains reduce how much a site gets crawled, which is why performance and clean URLs matter for discovery.