• About
    • History of Dallas SEO
    • SEO Expert Witness Services
  • Contact
  • Topics
    • Bing
    • Blogging
    • Branding
    • Domain Names
    • Google
    • Internet Marketing
    • Link Building
    • Local Search
    • Marketing
    • Public Relations
    • Reputation Management
    • Search Engine Marketing
    • Search Engine Optimization
    • Search Engines
    • Social Media
    • Tech
  • Advertise
  • Email Newsletter

Bill Hartzer

Bill Hartzer on Search, Marketing, Tech, and Domains.

SEMrush

Home » Google » Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Posted on May 17, 2019 Written by Bill Hartzer

backpack search result

A website’s home page is ranking very well for a competitive keyword despite the fact that the website is blocking the search engine crawlers from indexing the site. A search for the keyword “backpack” in Google shows a website’s home page, GSCBackpack.com, ranking #2 for the keyword. However, Google isn’t allowed to crawl the website (especially the site’s home page), and Google is ranking the keyword anyway.

Several years ago, Google, in what I can only describe as “overzealousness”, made a decision to start indexing web pages despite the website owners telling the search engines that they did NOT want their website (or web page) in the search results. Google decided to take a more “technical” approach to the matter. They made the decision that if you tell the search engines not to crawl a website, Google will index the page anyway. In my opinion, if a website owner tells a web crawler to NOT crawl their website, then the page or website shoudln’t be in the search engine results. The page should be removed, in my opinion, and the page shouldn’t show up in the search results at all–even if it’s for a search query like the word “backpack”.

Previously, if you told the search engines to not crawl your website, that meant that your website or a web page wouldn’t show up in the search results, as it cannot be crawled. Now, even if you tell them to stay out, not to crawl, your web page can still show up in the search results, like this case here, for the keyword “backpack”. Google took it upon themselves to change the rules.

Now, if you do NOT want a web page to appear in the Google search results, you actually must ALLOW the search engines to crawl your web page. I know, that doesn’t make sense. But you must allow them to crawl the web page so that they can see the “noindex” meta tag that you’ll put on your page.

If the search engines, mainly Google, have actually indexed a web page that you don’t want them to index, then you can remove the page from Google.

In the case of the keyword “backpack”, and this page still showing up in Google’s search engine results, the site owner is blocking the search engines from crawling their home page. They’re most likely mistakenly blocking them, and, as a result, the website is still ranking well, but the search result listing shows that the search engines are not allowed to crawl it.

This is the website’s robots.txt file as I currently see it:

robots.txt flile

As you can see, the * on the directive is causing this. I assume this is a mistake by the site owner, and will most likely be fixed or updated. The robots.txt file (all sites should have one) typically appears at domain.com/robots.txt .

Filed Under: Google

SEMrush

About Bill Hartzer

Bill Hartzer is CEO of Hartzer Consulting, an SEO Consulting firm that includes services such as search engine optimization, technical SEO audits, domain name consulting, and online reputation management. As an SEO Expert, Mr. Hartzer frequently serves as an SEO Expert Witness and Domain Name Expert Witness in legal cases worldwide.

Recent Posts

  • ChatGPT Versus Google Bard: Which is Better? March 22, 2023
  • All Domain Names Need SSL: Parked Domains Are Losing Traffic, Revenue March 17, 2023
  • Google is Finally Sunsetting Google Glass March 16, 2023
  • Microsoft Teams Free Classic Shutting Down April 12, 2023 February 21, 2023
  • GoDaddy Customer Loses Domain Name Due to Auto Renew Fail February 9, 2023
  • dotDB is Not Shutting Down February 1, 2023
  • Someone Stole My Domain Name: Here’s What You Do January 4, 2023
  • Web Hosting Services Market to Grow to $254.86 Billion by 2029 December 13, 2022
  • This SEO Blog Post Was Written by ChatGPT December 8, 2022
  • Facebook Rolling Out Facebook Articles December 7, 2022
  • Doing SEO is Better Than… December 6, 2022
  • Tucows and GoDaddy Report Q3 2022 Results November 6, 2022
  • How to Measure App Events Sourced by Organic Search and SEO September 20, 2022
  • Google Allegedly Eavesdrops and Monitors the Brain 24 hours a Day to Control Humanity September 14, 2022
  • Why You Shouldn’t Hire SEOs Based on An Email September 13, 2022
  • Global SEO Market to Reach $122.11 Billion by 2028 September 9, 2022
  • Bluehost Launches New Commerce Solutions for WordPress September 8, 2022
  • Which CMS? How to Choose the Best CMS for Your Purposes August 29, 2022
  • Accidental SEO Manager: Interview with Ash Nallawalla August 15, 2022
  • Sometimes Google Isn’t Family Friendly August 1, 2022

US Agency Awards Judge

DFWSEM logo

Bill Hartzer is a Brand Ambassador for:



Industry Friends

I Love SEO
WTFSEO
SEO By the Sea
Jeff Lenney
Jeff Gabriel
Phil Drinkwater
Dixon Jones
Brian Hartzer
Navah Hopkins
DNAccess

Connect With Bill Hartzer

Bill Hartzer on Twitter
Bill Hartzer on Instagram
Hartzer Consulting on Facebook
Bill Hartzer on Facebook
Bill Hartzer on YouTube

Categories

  • Advertising (19)
  • Bing Search Engine (6)
  • Blogging (42)
  • Branding (12)
  • Domain Names (212)
  • Google (237)
  • Internet Marketing (25)
  • Internet Usage (85)
  • Link Building (53)
  • Local Search (39)
  • Marketing (182)
  • Marketing Foo (30)
  • Pay Per Click (3)
  • Podcast (18)
  • Public Relations (8)
  • Reputation Management (9)
  • Search Engine Marketing (44)
  • Search Engine Marketing Events (48)
  • Search Engine Marketing Firms (19)
  • Search Engine Marketing Jobs (33)
  • Search Engine Optimization (164)
  • Search Engines (204)
  • Social Media (192)
  • Tech (7)
  • Web Analytics (17)
  • Webinars (1)

Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only, and are mentioned only to help my readers. All other trademarks cited herein are the property of their respective owners. Use of these names, logos, and brands does not imply endorsement.




Hartzer Consulting



Website, Content, and Marketing by Hartzer Consulting, LLC.

Copyright © 2023 ·