• About
    • History of Dallas SEO
    • SEO Expert Witness Services
  • Contact
  • Topics
    • Bing
    • Blogging
    • Branding
    • Domain Names
    • Google
    • Internet Marketing
    • Link Building
    • Local Search
    • Marketing
    • Public Relations
    • Reputation Management
    • Search Engine Marketing
    • Search Engine Optimization
    • Search Engines
    • Social Media
    • Tech
  • Advertise
  • Email Newsletter

Bill Hartzer

Bill Hartzer on Search, Marketing, Tech, and Domains.

SEMrush

Home » Google » Google’s Googlebot Causes United Airlines Stock to Plummet Says Tribune Company

Google’s Googlebot Causes United Airlines Stock to Plummet Says Tribune Company

Posted on September 10, 2008 Written by Bill Hartzer

Apparently Google’s search agent named Googlebot (their web crawler) caused the stock of United Airlines to plummet on September 7, 2008.

The Tribune Company, in a press released late today, said that “the confusion surrounding a 2002 Chicago Tribune article on the Internet this past weekend started with the inability of Google’s automated search agent “Googlebot” to differentiate between breaking news and frequently viewed stories on the websites of its newspapers.”

Apparently The Tribune Company has identified problems with Googlebot several months ago and they have asked Google to stop using Googlebot to crawl newspaper websites, including The Sun Sentinel (Ft. Lauderdale), for inclusion in Google News. But, the company says that even though they have requested that Googlebot stop crawling, they continued to crawl The Sun Sentinel’s website. Furthermore, the Tribute company believes “that Googlebot continues to misclassify stories.”

Let’s get one thing straight here. I am pretty familiar with Google, Google Webmaster Tools, and the technology used behind web sites. I’ve been doing search engine optimization since 1996. Essentially, if the Tribune Company was to verify their site in Google Webmaster Tools (which it appears that they might have done already), they should be able to stop Google’s bots from crawling their site. Furthermore, it is my opinion that if they were to also have a better robots.txt file on the web site they might be able to further control the crawling of Googlebot. Also, on the back-end of a web site it is possible to identify Googlebot and literally stop them from crawling the site.

The Tribune Company has released a summary of the sequence of events that apparently was started by Googlebot’s crawling The Sun Sentinel’s website in the late-evening and early- morning hours of September 6 and September 7. The summary is as follows:

The article, headlined “United Airlines Files for Bankruptcy,” was originally published in the Chicago Tribune in 2002, and appeared on the newspaper’s website. It then became part of the online database of Tribune’s newspapers. Our records indicate that the Googlebot crawled this story as recently as September 2 and September 3 and apparently treated it as old news.

On September 7, 2008 at 1:00:34 ET, (Sept. 6, 2008, 10:00:34 PT) our records indicate that the article received a single visit. Given the fact that it was the middle of the night, traffic to the business section of the Sun Sentinel site was very low at the time. We believe that this single visit resulted in a link to the old article being created on a dynamic portion of the Sun Sentinel’s business section under a tab called “Popular Stories Business: Most Viewed.”

Again, no new story was published and the old story was not re-published-a link to the old story was merely created. The URL for the old story did not change when the link appeared.

On September 7, at 1:36:03 ET (Sept. 6, 10:36:03) a user of the Sun Sentinel’s website, viewing a story about airline policies regarding cancelled flights, clicked on the link to the old story under the “Popular Stories Business: Most Viewed” tab. Fifty-two seconds later, at 1:36:57 ET (10:36:57 PT), Googlebot visited the Sun Sentinel’s website again and crawled the story.

This time, despite the fact that the URL to the old story hadn’t changed, despite the fact that Googlebot had seen this story previously, it was apparently treated as though it was breaking news. Shortly thereafter, Google provided a link to the old story on Google News and dated it September 6, 2008. Google’s dating the story on Google News made it appear current to Google News users.

The first referral to the story from the link provided by Google News came just three minutes later, at 1:39:59 ET (10:39:59 PT).

Traffic to the old story increased during the course of the day, Sunday, September 7, with the bulk of it being referrals from Google. On Monday, September 8, traffic increased even more after a summary of the Google News story was made available to subscribers of Bloomberg News.

So, from what the Tribune Company is saying, although Google News had previously published this story several years ago, Google News treated this story as if it were breaking news. Not only that, Google News continued to make the story available to Bloomberg News subscribers.

Let me just ask this basic question: Do you believe everything that you hear and read on the Internet? Can we assume that everything in the news is true and correct?

Update 9/11/2008: BlogStorm has written a great post about what happened, and points out all of the duplicate content that might have caused the issue in the first place.

Also, you might want to take a look at Google’s explanation of what happened and why Google News was led to believe that this was a new story and not an old one.

Filed Under: Google

SEMrush

About Bill Hartzer

Bill Hartzer is CEO of Hartzer Consulting, an SEO Consulting firm that includes services such as search engine optimization, technical SEO audits, domain name consulting, and online reputation management. As an SEO Expert, Mr. Hartzer frequently serves as an SEO Expert Witness and Domain Name Expert Witness in legal cases worldwide. He also oversees DNAccess.com, a company that provides brand protection and monitoring, domain name background checks, and stolen domain name recovery services.

Recent Posts

  • Google Search Engine Class Action Settlement: Google Referrer Header Privacy Litigation June 5, 2023
  • I Asked Twitter for SEO Advice. Here’s What They Said June 1, 2023
  • Web.com Launches AI Domain Name Generator and AI Writer Tools May 31, 2023
  • Artlist Launches AI-Powered Footage Search Engine May 30, 2023
  • Google’s Search Generative Experience According to Google Bard May 22, 2023
  • Is AIO the New SEO? May 10, 2023
  • Google Passkey Support to Replace Built-In Security Key for Google Accounts May 3, 2023
  • WisTL Launches WisTLing App to Explore The Filming Locations of TV Series and Movies April 28, 2023
  • BrandChamp Acquires CrewFire April 12, 2023
  • Facebook Won’t Verify My Identity for Meta Verified April 4, 2023
  • ChatGPT Versus Google Bard: Which is Better? March 22, 2023
  • All Domain Names Need SSL: Parked Domains Are Losing Traffic, Revenue March 17, 2023
  • Google is Finally Sunsetting Google Glass March 16, 2023
  • Microsoft Teams Free Classic Shutting Down April 12, 2023 February 21, 2023
  • GoDaddy Customer Loses Domain Name Due to Auto Renew Fail February 9, 2023
  • dotDB is Not Shutting Down February 1, 2023
  • Someone Stole My Domain Name: Here’s What You Do January 4, 2023
  • Web Hosting Services Market to Grow to $254.86 Billion by 2029 December 13, 2022
  • This SEO Blog Post Was Written by ChatGPT December 8, 2022
  • Facebook Rolling Out Facebook Articles December 7, 2022

Hartzer Domains

US Agency Awards Judge

DFWSEM logo

Bill Hartzer is a Brand Ambassador for:



Industry Friends

I Love SEO
WTFSEO
SEO By the Sea
Jeff Lenney
Jeff Gabriel
Phil Drinkwater
Dixon Jones
Brian Hartzer
Navah Hopkins
DNAccess
SEO Dallas

Connect With Bill Hartzer

Bill Hartzer on Twitter
Bill Hartzer on Instagram
Hartzer Consulting on Facebook
Bill Hartzer on Facebook
Bill Hartzer on YouTube

Categories

  • Advertising (19)
  • AI (1)
  • Bing Search Engine (6)
  • Blogging (42)
  • Branding (12)
  • Domain Names (213)
  • Google (240)
  • Internet Marketing (25)
  • Internet Usage (85)
  • Link Building (53)
  • Local Search (39)
  • Marketing (183)
  • Marketing Foo (30)
  • Pay Per Click (3)
  • Podcast (18)
  • Public Relations (8)
  • Reputation Management (9)
  • Search Engine Marketing (44)
  • Search Engine Marketing Events (48)
  • Search Engine Marketing Firms (19)
  • Search Engine Marketing Jobs (33)
  • Search Engine Optimization (165)
  • Search Engines (205)
  • Social Media (194)
  • Tech (7)
  • Web Analytics (17)
  • Webinars (1)

Note: All product names, logos, and brands are property of their respective owners. All company, product and service names used in this website are for identification purposes only, and are mentioned only to help my readers. All other trademarks cited herein are the property of their respective owners. Use of these names, logos, and brands does not imply endorsement.




Hartzer Consulting



Website, Content, and Marketing by Hartzer Consulting, LLC.

Copyright © 2023 ·