• About
    • History of Dallas SEO
  • Contact
  • Topics
    • Bing
    • Blogging
    • Branding
    • Domain Names
    • Google
    • Internet Marketing
    • Link Building
    • Local Search
    • Marketing
    • Public Relations
    • Reputation Management
    • Search Engine Marketing
    • Search Engine Optimization
    • Search Engines
    • Social Media
    • Tech
  • Advertise
  • Services
    • Search Engine Optimization
    • Ongoing SEO Services
    • SEO Expert Witness
    • Google Penalty Recovery
    • Mini SEO Audit
    • Link Audit
    • Keyword Research
    • Combine Websites SEO Services
    • PPC Management
    • Online Reputation Management
    • Domain Name Consultant
    • Domain Names & Expired Domains
    • Domain Name Appraisal

Bill Hartzer

GoDaddy Airo: Register your .com domain name today!
Home » Google » Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Page Blocked by Robots.txt Still Ranks for Competitive Keyword

Posted on May 17, 2019 Written by Bill Hartzer

backpack search result

A website’s home page is ranking very well for a competitive keyword despite the fact that the website is blocking the search engine crawlers from indexing the site. A search for the keyword “backpack” in Google shows a website’s home page, GSCBackpack.com, ranking #2 for the keyword. However, Google isn’t allowed to crawl the website (especially the site’s home page), and Google is ranking the keyword anyway.

Several years ago, Google, in what I can only describe as “overzealousness”, made a decision to start indexing web pages despite the website owners telling the search engines that they did NOT want their website (or web page) in the search results. Google decided to take a more “technical” approach to the matter. They made the decision that if you tell the search engines not to crawl a website, Google will index the page anyway. In my opinion, if a website owner tells a web crawler to NOT crawl their website, then the page or website shoudln’t be in the search engine results. The page should be removed, in my opinion, and the page shouldn’t show up in the search results at all–even if it’s for a search query like the word “backpack”.

Previously, if you told the search engines to not crawl your website, that meant that your website or a web page wouldn’t show up in the search results, as it cannot be crawled. Now, even if you tell them to stay out, not to crawl, your web page can still show up in the search results, like this case here, for the keyword “backpack”. Google took it upon themselves to change the rules.

Now, if you do NOT want a web page to appear in the Google search results, you actually must ALLOW the search engines to crawl your web page. I know, that doesn’t make sense. But you must allow them to crawl the web page so that they can see the “noindex” meta tag that you’ll put on your page.

If the search engines, mainly Google, have actually indexed a web page that you don’t want them to index, then you can remove the page from Google.

In the case of the keyword “backpack”, and this page still showing up in Google’s search engine results, the site owner is blocking the search engines from crawling their home page. They’re most likely mistakenly blocking them, and, as a result, the website is still ranking well, but the search result listing shows that the search engines are not allowed to crawl it.

This is the website’s robots.txt file as I currently see it:

robots.txt flile

As you can see, the * on the directive is causing this. I assume this is a mistake by the site owner, and will most likely be fixed or updated. The robots.txt file (all sites should have one) typically appears at domain.com/robots.txt .

Filed Under: Google

About Bill Hartzer

Bill Hartzer is the CEO of Hartzer Consulting and founder of DNAccess, a domain name protection and recovery service. A recognized authority in digital marketing and domain strategy, Bill is frequently called upon as an Expert Witness in internet-related legal cases. He's been sharing insights and research here on BillHartzer.com for over two decades.

Bill Hartzer on Search, Marketing, Tech, and Domains.

Recent Posts

  • Unbounce Doubles Down on AI and Automation to Boost Go-To-Market Teams July 10, 2025
  • NameSilo to Acquire SewerVUE Technology in $2.45M Deal July 10, 2025
  • Internet Marketing Ninjas Acquired by Previsible.IO July 9, 2025
  • Metricool Brings Real Analytics to Personal LinkedIn Profiles July 8, 2025
  • This Cleveland Agency Found a Smarter Way to Rank in Every Suburb—Without Opening More Offices July 8, 2025
  • Survey: Gen Z Reuses Passwords but Demands Bank-Level Security From Small Businesses July 8, 2025
  • Liftoff Reveals What’s Actually Working in Mobile Ads July 7, 2025
  • EasySend’s Big Move: AI Tools That Make Static Forms Obsolete July 7, 2025
  • Is Social Media Failing Small Businesses? New Survey Reveals a Hidden Blind Spot July 7, 2025
  • Why Cloudflare’s Pay Per Crawl Is a Trap for 99% of Websites July 2, 2025
  • The Hidden Risk of Double Letters in Brand and Domain Names July 2, 2025
  • GEO Verified™ Launches to Help Brands Survive the AI Search Shakeup July 1, 2025
  • RetailOnline.com Hits the Market After 25 Years—And It’s Built for the Future of E-Commerce July 1, 2025
  • AI-Powered Task Planning: The Future of Business Efficiency and Personal Productivity June 30, 2025
  • New Yoast Add-On Turns Google Docs Into an SEO Power Tool June 26, 2025
  • Simon Data Flips the Script on Marketing with AI Agents June 26, 2025
  • IAB Lays Down the Law for Gaming Ads—Here’s What Brands Need to Know June 26, 2025
  • Google Review Extortion Text Message – Scam Warning for Business Owners June 25, 2025
  • Google Names SearchKings Top AI Innovator for Transforming Lead Quality June 24, 2025
  • Marketing Exec Buys Social Media Firm in Deal That Signals Big Plans June 24, 2025

Hartzer Domains

Bare-Metal Servers by HostDime

DFWSEM logo

Bill Hartzer is a Brand Ambassador for:

Industry Friends

I Love SEO
WTFSEO
SEO By the Sea
Brian Harnish
Jeff Lenney
Jeff Gabriel
Scott Hendison
Dixon Jones
Brian Hartzer
Navah Hopkins
DNAccess
SEO Dallas
Confirmed Stolen

Connect With Bill Hartzer

Bill Hartzer on Twitter
Bill Hartzer on BlueSky
Bill Hartzer on Instagram
Hartzer Consulting on Facebook
Bill Hartzer on Facebook
Bill Hartzer on YouTube

Categories

  • Advertising (109)
  • AI (201)
  • Bing Search Engine (8)
  • Blogging (43)
  • Branding (19)
  • Domain Names (316)
  • Google (260)
  • Internet Marketing (52)
  • Internet Usage (95)
  • Link Building (53)
  • Local Search (63)
  • Marketing (232)
  • Marketing Foo (34)
  • Pay Per Click (9)
  • Podcast (19)
  • Public Relations (9)
  • Reputation Management (14)
  • Search Engine Marketing (46)
  • Search Engine Marketing Events (60)
  • Search Engine Marketing Firms (94)
  • Search Engine Marketing Jobs (33)
  • Search Engine Optimization (189)
  • Search Engines (223)
  • Social Media (302)
  • Social Media Marketing (58)
  • Tech (16)
  • Web Analytics (21)
  • Webinars (1)

Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only, and are mentioned only to help my readers. All other trademarks cited herein are the property of their respective owners. Use of these names, logos, and brands does not imply endorsement.

 

Hartzer Consulting

Website, Content, and Marketing by Hartzer Consulting, LLC.

Disclaimer - Privacy Policy - Terms of Use

Copyright © 2025 ·