.Google.com's John Mueller responded to an inquiry concerning why Google indexes web pages that are actually refused coming from creeping through robots.txt and also why the it is actually safe to overlook the related Browse Console documents about those creeps.Crawler Traffic To Concern Criterion URLs.The individual talking to the concern chronicled that crawlers were developing web links to non-existent concern parameter Links (? q= xyz) to webpages with noindex meta tags that are additionally blocked in robots.txt. What caused the concern is that Google.com is creeping the links to those web pages, receiving shut out by robots.txt (without envisioning a noindex robots meta tag) then obtaining reported in Google.com Search Console as "Indexed, though obstructed by robots.txt.".The individual talked to the complying with concern:." Yet listed below is actually the large question: why would certainly Google.com index webpages when they can not even see the material? What is actually the advantage because?".Google's John Mueller affirmed that if they can not creep the webpage they can not find the noindex meta tag. He also produces an intriguing acknowledgment of the web site: hunt operator, advising to dismiss the end results considering that the "ordinary" users will not view those results.He wrote:." Yes, you are actually proper: if our team can not creep the webpage, our team can't find the noindex. That mentioned, if our team can not creep the pages, then there's certainly not a lot for our company to index. So while you may see a few of those pages with a targeted site:- question, the typical customer won't find all of them, so I wouldn't bother it. Noindex is actually additionally fine (without robots.txt disallow), it simply indicates the Links will find yourself being actually crept (and find yourself in the Browse Console report for crawled/not indexed-- neither of these statuses create problems to the remainder of the web site). The important part is that you do not produce them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limits in operation the Site: search evolved hunt driver for diagnostic causes. Among those explanations is actually because it is actually not linked to the routine hunt index, it is actually a separate trait altogether.Google.com's John Mueller commented on the web site search driver in 2021:." The short solution is actually that a web site: concern is certainly not indicated to be complete, nor made use of for diagnostics purposes.A website query is a certain kind of search that confines the outcomes to a particular website. It is actually primarily merely words web site, a digestive tract, and then the web site's domain name.This query limits the results to a details site. It's certainly not indicated to be an extensive assortment of all the webpages from that internet site.".2. Noindex tag without using a robots.txt is actually great for these type of scenarios where a crawler is connecting to non-existent webpages that are actually obtaining uncovered through Googlebot.3. Links along with the noindex tag are going to generate a "crawled/not catalogued" entry in Search Console which those will not have a negative result on the rest of the website.Check out the question and also address on LinkedIn:.Why would Google mark pages when they can't even observe the material?Included Image through Shutterstock/Krakenimages. com.