• About
    • History of Dallas SEO
    • SEO Expert Witness Service
  • Contact
  • Topics
    • Bing
    • Blogging
    • Branding
    • Domain Names
    • Google
    • Internet Marketing
    • Link Building
    • Local Search
    • Marketing
    • Public Relations
    • Reputation Management
    • Search Engine Marketing
    • Search Engine Optimization
    • Search Engines
    • Social Media
    • Tech
  • Advertise
  • Email Newsletter

Bill Hartzer

Bill Hartzer on Search, Marketing, Tech, and Domains.

Semrush Free Trial

Home » Google » Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Posted on May 17, 2019 Written by Bill Hartzer

backpack search result

A website’s home page is ranking very well for a competitive keyword despite the fact that the website is blocking the search engine crawlers from indexing the site. A search for the keyword “backpack” in Google shows a website’s home page, GSCBackpack.com, ranking #2 for the keyword. However, Google isn’t allowed to crawl the website (especially the site’s home page), and Google is ranking the keyword anyway.

Several years ago, Google, in what I can only describe as “overzealousness”, made a decision to start indexing web pages despite the website owners telling the search engines that they did NOT want their website (or web page) in the search results. Google decided to take a more “technical” approach to the matter. They made the decision that if you tell the search engines not to crawl a website, Google will index the page anyway. In my opinion, if a website owner tells a web crawler to NOT crawl their website, then the page or website shoudln’t be in the search engine results. The page should be removed, in my opinion, and the page shouldn’t show up in the search results at all–even if it’s for a search query like the word “backpack”.

Previously, if you told the search engines to not crawl your website, that meant that your website or a web page wouldn’t show up in the search results, as it cannot be crawled. Now, even if you tell them to stay out, not to crawl, your web page can still show up in the search results, like this case here, for the keyword “backpack”. Google took it upon themselves to change the rules.

Now, if you do NOT want a web page to appear in the Google search results, you actually must ALLOW the search engines to crawl your web page. I know, that doesn’t make sense. But you must allow them to crawl the web page so that they can see the “noindex” meta tag that you’ll put on your page.

If the search engines, mainly Google, have actually indexed a web page that you don’t want them to index, then you can remove the page from Google.

In the case of the keyword “backpack”, and this page still showing up in Google’s search engine results, the site owner is blocking the search engines from crawling their home page. They’re most likely mistakenly blocking them, and, as a result, the website is still ranking well, but the search result listing shows that the search engines are not allowed to crawl it.

This is the website’s robots.txt file as I currently see it:

robots.txt flile

As you can see, the * on the directive is causing this. I assume this is a mistake by the site owner, and will most likely be fixed or updated. The robots.txt file (all sites should have one) typically appears at domain.com/robots.txt .

Filed Under: Google



Listen to "Digital Marketing with Bill Hartzer" on Spreaker.

About Bill Hartzer

Bill Hartzer is CEO of Hartzer Consulting, LLC, an SEO Consulting firm that includes services such as search engine optimization, technical SEO audits, domain name consulting, and online reputation management.

Recent Posts

  • Sometimes Google Isn’t Family Friendly August 1, 2022
  • Something’s Seriously Wrong with Facebook Notifications July 12, 2022
  • Facebook Internet Tracking Settlement June 24, 2022
  • RankSense Acquired by SEOClarity June 1, 2022
  • LinkedIn Links, Digital Marketing News, and SEO Questions Answered May 9, 2022
  • GoDaddy Ending Forwarding of Existing Shortened Links May 5, 2022
  • 7 Ways to Protect Your Domain Name March 31, 2022
  • Majestic Withdrawing from the Russian Market March 14, 2022
  • Sedo Suspends Parking for Domains Under .RU and .BY March 10, 2022
  • Someone Stole My Domain Name: Here’s What You Do March 7, 2022
  • Stolen Domain Name Patterns.com Returned to Owner March 1, 2022
  • Namecheap Terminating Services to Russian Customers February 28, 2022
  • Getting Started with TikTok Marketing for Your Business with Austin Armstrong February 17, 2022
  • How I Got 100 Million Views On My Google Maps Photos February 15, 2022
  • Is Google Concerned About Being Regulated? February 8, 2022
  • Podcast: Social Media Previews, Domain Name TLD Zone Files, and How Thieves Steal Domain Names January 21, 2022
  • Google Cameos Shutting Down February 17, 2022 January 21, 2022
  • Navah Hopkins Launches Navah Hopkins Consulting January 13, 2022
  • Class Action Lawsuit: Google is Paying Apple to Stay Out of Search Business January 5, 2022
  • SE Ranking Launches SE Ranking Academy December 28, 2021

DFWSEM logo

Bill Hartzer is a Brand Ambassador for:



Industry Friends

I Love SEO
WTFSEO
SEO By the Sea
Jeff Lenney
Jeff Gabriel
Phil Drinkwater
Dixon Jones
Brian Hartzer
Navah Hopkins

Connect With Bill Hartzer

Bill Hartzer on Twitter
Bill Hartzer on Instagram
Hartzer Consulting on Facebook
Bill Hartzer on Facebook
Bill Hartzer on YouTube

Categories

  • Advertising (19)
  • Bing Search Engine (6)
  • Blogging (42)
  • Branding (12)
  • Domain Names (208)
  • Google (235)
  • Internet Marketing (25)
  • Internet Usage (84)
  • Link Building (53)
  • Local Search (39)
  • Marketing (178)
  • Marketing Foo (30)
  • Pay Per Click (3)
  • Podcast (17)
  • Public Relations (8)
  • Reputation Management (9)
  • Search Engine Marketing (44)
  • Search Engine Marketing Events (48)
  • Search Engine Marketing Firms (19)
  • Search Engine Marketing Jobs (33)
  • Search Engine Optimization (160)
  • Search Engines (204)
  • Social Media (191)
  • Tech (7)
  • Web Analytics (16)
  • Webinars (1)




Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only, and are mentioned only to help my readers. All other trademarks cited herein are the property of their respective owners. Use of these names, logos, and brands does not imply endorsement.




Hartzer Consulting



Website, Content, and Marketing by Hartzer Consulting, LLC.

Copyright © 2022 ·