Robots.txt Metrics

To help you quickly detect if certain URLs are blocked by robots.txt, we provide detailed robot.txt metrics for every link.

Robots.txt Bots txt allows general bots for currents URL
Robots.txt Googlebot txt allows Googlebot for current URL
Robots.txt Allowed Bots Major Bots* are allowed by Robots.txt for current URL
Robots.txt Blocked Bots Major Bots are blocked by Robots.txt for current URL

*major Bots = applebot, baiduspider, bingbot, googlebot, msnbot, naverbot, seznambot, slurp, teoma, yandex

All these metrics are hidden by default. You can see all of them in your report by clicking on: “show all columns”, or you can just click on the ones you need.