robots.txt and Googlebot: how a blocked crawl kills your application

robots.txt is a plain text file at the root of your domain that tells compliant crawlers which paths they may request. It controls crawling, not indexing, and it is advisory: well-behaved crawlers obey it, malicious ones ignore it. It is not a security mechanism, and anything you disallow is still publicly reachable by anyone who knows the URL.

For an AdSense application it matters for one reason: if Googlebot is disallowed, Google cannot read your content, and a site whose content cannot be read cannot demonstrate value.

• Googlebot — the search crawler. This is what indexes your pages and what a content review depends on.
• Mediapartners-Google — the AdSense crawler. It reads pages to decide which ads are relevant. Blocking it does not stop ads from serving, but it makes them less relevant and therefore less valuable.

Both should be allowed across your public pages. There is no reason for a monetised content site to block either.

• User-agent: *
• Allow: /
• Sitemap: https://example.com/sitemap.xml

That is the whole thing for most sites. Add specific Disallow lines only for paths that genuinely should not be crawled — admin areas, internal search result pages, checkout steps, endless filter permutations.

• Disallow: / under User-agent: * — blocks the entire site. This is the standard staging-site file, and it ships to production more often than anyone admits.
• A block specifically naming User-agent: Googlebot with Disallow: / — usually copied from a tutorial about blocking a misbehaving bot.
• Disallow: /blog/ or /guides/ on the exact directory holding your content, left over from a rebuild.
• Disallow: /*.js$ or /*.css$ — Google needs to render your pages; blocking assets makes a modern site look broken to the crawler and can affect mobile-usability judgements.
• Noindex: directives inside robots.txt. Google stopped supporting these years ago; they do nothing, and people who rely on them are often surprised in the opposite direction.

Two rules trip people up. First, a crawler obeys exactly one group: the one whose user-agent token most specifically matches its own name. If there is a Googlebot group, Googlebot reads that group and completely ignores the wildcard group, including any Allow lines in it. Second, within the matching group, the most specific rule wins — a longer path pattern beats a shorter one, and on an exact tie Allow beats Disallow.

This is why adding a friendly Allow: / to the wildcard group does not rescue a Disallow: / sitting in a Googlebot-specific group above it. Fix the group that applies, not a different one.

robots.txt is only half the story. A page can be perfectly crawlable and still carry a meta robots noindex tag in its head, or an X-Robots-Tag: noindex header from the server. Those genuinely remove the page from search results. Most CMSs have a 'discourage search engines' switch that sets exactly this, and forgetting to turn it off after launch is one of the most common reasons a finished site has no indexed pages.

A subtle trap: if you block a URL in robots.txt, Google cannot fetch it, so it cannot see a noindex tag on it either. To remove a page from search, allow the crawl and use noindex — never both.

1. Open https://yourdomain.com/robots.txt in a private window and read it. If you get a 404, you have no file — that is fine and means everything is crawlable.
2. Check for any Disallow: / and for any group naming Googlebot.
3. View the source of a content page and search for 'noindex'.
4. In Search Console, use URL Inspection on a real content URL and confirm it reports the page as indexable.
5. Run the domain through an automated checker for a second opinion on the file's presence, status code and Googlebot rules.