Skip to content
whois.me.uk
Menu
Sign in

AI-ROBOTS-UNREADABLE

robots.txt could not be read

What it means

Crawlers that cannot fetch robots.txt fall back to their own defaults, which for most means crawling everything. A redirect loop, a 5xx error or a TLS problem on the robots.txt URL also delays indexing.

How to fix it

  1. 1Serve /robots.txt as text/plain with a 200 status on the canonical host.
  2. 2Avoid redirecting robots.txt to another host unless that host serves the file correctly.

Re-check after the change

DNS changes can take up to the record's TTL to be seen; use Re-run on the result page to bypass our 10-minute cache.