Robots.txt Checker

What the robots.txt checker tests

Checks

robots.txt availability and sitemap directive presence
Allow/Disallow rules for Googlebot, Bingbot, and wildcard User-agent
whether a specific path is blocked because of incorrect user-agent grouping

Example result

For /tools/favicon/, the checker can show Googlebot as allowed, YandexBot as blocked, and a sitemap URL discovered in robots.txt.

Common mistakes

Disallow: / is accidentally applied to User-agent: *
/api or /_next is blocked without allowing needed static assets
robots.txt blocks an important URL that still appears in the sitemap

Who uses it

SEO specialists
developers after robots.txt edits
site owners after migrations

Robots.txt FAQ

Does robots.txt guarantee deindexing?

No. Robots.txt controls crawling. For deindexing, use noindex on a page that crawlers can access.

Why do different bots get different rules?

Robots.txt is grouped by User-agent. A rule for Yandex should not automatically apply to Googlebot.