The AdSense control panel has a new page – “Site Diagnostics”. This page lists URLs on your sites that the AdSense crawler was not able to access. All AdSense publishers should check this page to make sure that all of their pages are being fully monetized.
The reasons listed for the crawler not being able to access pages are:
- Robots.txt file
- URL not found
Found via ProBlogger