Facebook Acting More and More Like a Search Engine

Is it just me, or is Facebook acting more and more like a search engine, crawling all sorts of URLs. More than they have in the past. Facebook is acting more and more like a search engine every single day. Here is a perfect example:

Facebook bot visits

If you take a look at the screen capture above from my stats just now, you’ll notice that Facebook Bot is visiting over and over again. But not only are they visiting over and over again, that part I can possibly understand. If someone posts a link in Facebook to a URL then it would make sense that Facebook would visit and check out that URL just to keep Facebook “clean” so to speak. So that there are not any bad links floating around Facebook.

But, in this case, Facebook Bot is visiting FROM a LinkedIn URL, and they are visiting the site FROM a Twitter link. They’re actually crawling, it appears, and the referrer is from Twitter and from LinkedIn.

In this case, I do not believe that people posted the LinkedIn redirect URL in Facebook. I don’t think someone posted the actual Twitter URL in Facebook. If someone had done that, the URL would technically show a referrer from Facebook.com or m.facebook.com, which is showing up in other referrers in my stats.

So, this appears to be Facebook Bot crawling links. Right now I’m seeing referrers from LinkedIn and from Twitter. I haven’t seen any Google+ URls show up (yet) where Facebook Bot is crawling Google URLs.

So, is Facebook Bot really Facebook’s new search engine? Are they crawling? What do you think?

Study Google+ Ripples to See Why Posts Go Hot

Just recently, another one of my Google+ posts went hot on the Google Plus social media network. As you know, Google Authorship is tied directly to Google+, and it’s my opinion that the more of an influencer that you are on Google+ the more it will ultimately help your Google Authorship. And then when you post on your own site or even post on other sites your content will ultimately do better and be seen by more people if you’re active on Google Plus.

Take a look at my latest post that got to the “What’s Hot” page on Google Plus:

Google Plus What's Hot

What’s not really important here is the actual content. Sure, it’s an interesting photo, and we can debate all day long why this particular photo ultimately got the number of +1s and how many comments and shares it got. I have a formula and a ‘method’ for getting something to “go hot”, and I’m not going to discuss it in this post right now.

But what’s more important here is to see WHY this post went hot and HOW it went hot. Well, how do you do that?

You can use Google Ripples to see exactly what, who, when, and why something went hot.

If you study the Google Ripples of posts that made it to the “What’s Hot” page, then you can learn a lot. Not only will you understand when something went hot, but you can see exactly who shared and reshared the post. And you can see exactly who the influencers are who made it go hot.

This is an incredible resource, especially if you are wanting to build up your following and connect with people who are the real influencers.

If you don’t know how to view the Google Ripples of a post, click on the link at the top right of the post. You will see a drop-down where you can “view ripples”.

Take a look at the Google Ripples for my post, which went hot yesterday:

google ripples

You can actually view who the influencers are by viewing the Google Ripples.

Google Not Provided Keeps Going Up

The data that Google is not providing, as in they are not providing which keyword brought visitors to your site, keeps going up. And it’s getting bad. Really bad if your site caters to the tech crowd. For example, for my blog on BillHartzer.com, the posts and articles on the site cater mostly to other webmasters and perhaps business owners. The more tech-savvy ones. And for my blog, the latest number of not provided users to hit the blog from Google organic search is now 71.80 percent (the past 20,000 visitors to the site via Google organic search).

Google Not Provided

That is way up over the average for this year is up nearly 20 percent! And to be honest with you, the general keywords that I’m ranking for as well as the overall topic of the blog has not changed.

Take a look at a comparison between January 1, 2013 and August 26, 2013 versus the previous period:

google not provided up

There are a few things going on here. I believe that people are logging into Google and thus performing https secure searches more often. Especially with more people using Android-powered phones and tablet. And to be honest with you, whenever I use my iPad and use a Google service, I’m logged in, as well. So my searches are secure: secure searches result in “not provided” being passed in the referrer.

But the other issue here is that Google is doing this, as well. Sure, more people are using a secure Google search. But is it really 20 percent more over the past year? Really?!?

Let’s take a look at a recent report released by BrightEdge:

brightedge not provided 49 percent

BrightEdge has released a report showing that for the 8,400 brands they tracked over the last quarter, 49% of the queries collected did not provide (i.e. not provided) search query data due to Google’s secure search.

49% was the average, in fact, it is higher for the technology industry.

Barry Schwartz even noticed that his average was over 70 percent. Which, in this case, is right in line with what mine is right now. I’m seeing 71.80 percent as being “not provided”.

So, what can you do if your “not provided” keeps going up? You’re supposed to get that data from Google Webmaster Tools. If you’ve verified your site with Google Webmaster Tools, log in and then click on the Search Queries link on the the left side. It will give you the data that is missing, or “not provided”. See below:

Google Webmaster Tools Search Queries

I can honestly tell you that it appears that the “not provided” data is really being reported in Google Webmaster Tools. For example, for one particular keywords, I’ll call it “red widgets”:

Visits (clicks) reported in Webmaster Tools: 60
Visits reported in Google Analytics: 16

So, it does look like the data is in Google Webmaster Tools. This is just one sample keyword, but looking at other keywords it’s reasonable to say that you do get the data in Webmaster Tools if you verify your site.

OOPS! Altavista Gone, But Only Temporarily?

altavista 1999

Altavsita, one of my favorite 1990s search engines, has been put out to pasture so to speak. Altavista.com now redirects to a page on Yahoo.com. But, mistakenly, it looks like Altavista is only temporarily being redirected to Yahoo!.

Whenever you take down a website, it should be redirected to its new location. So, if someone were to type in www.Altavista.com in their web browser, they should be redirected to the new website: in this case, to Yahoo’s website. That’s what Yahoo! has done here, they are now redirecting the old Altavista website to a page on Yahoo.com.

Here is a screen shot of the server headers as of the writing of this post:

altavista yahoo redirect

But, in this case, there is a twist. Technically speaking, Altavista is only being redirected temporarily to the Yahoo! location, not permanently, which is the “proper” way to set up a redirect. There are generally several different ways to redirect visitors to a new location or a new web URL:

- Add the meta “refresh” tag to the web page. In this case, you can add a certain number of seconds (a delay) so that you can notify users that they will be redirected to the new location. If you were to add a meta refresh of 10 seconds, you could then post a page on the old website telling people that there is a change, and that they will be redirected in a few seconds. Kind of the courtesy way to redirect users. But very “old school” and not recommended. In this case, no PageRank or “link juice” so to speak will be passed–not as much as using a 301 redirect. I don’t recommend ever using a meta refresh anymore.

- Add a 302 Moved Temporarily to the site. There are a few ways to do this, and even registrars like Godaddy offer to set up 302 redirect for you (or you can set it up on your web server). This will redirect users to the new location–but it’s only temporary so no search engine listings will be moved (the old URL will still be in the search engines’ indexes). This is generally not recommended. I would only set up a 302 Temporary Redirect if I need to redirect for a few days at most. No PageRank or “link juice” is typically passed through 302 redirects.

- Add a 301 Permanent Redirect to the site. This will redirect users to the new location. This is the preferred method, the recommended method by Google and even Bing.com. Bing populates Yahoo!’s search engine results. With a 301, users are redirected, PageRank and link juice is (mostly) passed to the new location. You set this type of redirect up on your web server (and even some registrars offer it as an option).

In this case, Altavista is being redirected with a 302 Temporary Redirect. So, either this means that it’s only temporarily been taken down (perhaps), or the person who set up the redirect does not know what they’re doing (most likely the scenario here). They may be only temporarily redirecting it, though, since it might be moving to a new location at some point.

In any case, though, I would not be using a 302 Redirect to redirect visitors from Altavista.com to Yahoo!. Or are they doing this on purpose?

FTC Sends Letter to Search Engines Regarding Paid Ads Versus Natural Search Results

The FTC has sent a letter to many of the major search engines, including AOL, Ask.com, Bing, Blekko, DuckDuckGo, Google, and Yahoo!, as well as 17 of the “most heavily trafficked search engines that specialize in the areas of shopping, travel, and local business, and that display advertisements to consumers” warning them about making the distinction between paid ads and natural search results. With all of the new ways that consumers can get their search results (desktop search, mobile search, voice search, etc.), the FTC is warning the search engines that consumers must be notified that paid ads are paid ads–and it must be very clear which search results include paid ads and which do not.

ftc guidance search engines

Originally, in 2002, the FTC revised its search engine guidance. Today’s letter to the search engines from the FTC begins:

“In 2002, the staff of the Federal Trade Commission’s Bureau of Consumer Protection published a letter (“2002 Search Engine Letter”) advising search engines about the potential for consumers to be deceived, in violation of Section 5 of the FTC Act, unless search engines clearly and prominently distinguished advertising from natural search results.

After the 2002 Search Engine Letter was issued, search engines embraced the letter’s guidance and distinguished any paid search results or other advertising on their websites. Since then, however, we have observed a decline in compliance with the letter’s guidance.”

The letter goes on to warn the search engines about the new ways consumers search the web:

the principles underlying the 2002 Search Engine letter remain the same: consumers ordinarily expect that natural search results are included and ranked based on relevance to a search query, not based on payment from a third party.

Including or ranking a search result in whole or in part based on payment is a form of advertising. To avoid the potential for deception, consumers should be able to easily distinguish a natural search result from advertising that a search engine delivers.

The FTC wants to be certain that the search engines make sure consumers are not deceived. It should be really easy to distinguish between paid ads and natural search results.

The FTC encourages search engines to use Visual Cues to distinguish paid ads versus natural search results:

we recommend that
in distinguishing any top ads or other advertising results integrated into the natural search results, search engines should use:
(1) more prominent shading that has a clear outline;
(2) a prominent border that distinctly sets off advertising from the natural search results;
or
(3) both prominent shading and a border.

An example of this would be the following search result:

FTC paid ads search engines

In the above screen shot, you’ll notice that Google distinguishes the paid ads versus the non-paid ads (natural search results) by shading the area of paid ads in yellow.

In addition to visual cues, the FTC wants text labels, as well, saying:

In addition to the visual cues a search engine may use to distinguish advertising, it also should have a text label that:
(1) uses language that explicitly and unambiguously conveys if a search result is advertising;
(2) is large and visible enough for consumers to notice it;
and
(3) is located near the search result (or group of search results) that it qualifies and where consumers will see it.

Google is complying with their recommendations by adding the text of “Ads related to concert tickets”, as seen in the screen capture below:

ads concert tickets

The FTC also mentions Font size:

Thus, we recommend that search engines place any text label used to distinguish advertising results immediately in front of an advertising result, or in the upper-left hand corner of an ad block, including any grouping of paid specialized results, in adequately sized and colored font.

The font size of that label saying it’s an ad must appear in front of the ad or in the upper left corner of an ad block.

Finally, the FTC addresses disclosure of paid ads in other forms, such as on social sites.

Regardless of the precise
form search may take in the future, the long-standing principle of making advertising distinguishable from natural results will remain applicable. For example, if a social network were to stream recommended restaurants based on what a particular consumer’s social contacts
have enjoyed, it should clearly distinguish as advertising any information feeds included or prioritized based in whole or in part on payments from a third party.

So, if you’re pulling ads from a feed, then you have to disclose that it’s an ad? I realize that this is going to be difficult in some cases–or maybe in a lot of cases. But, it’s a new requirement from the FTC. And, most interestingly, the FTC includes information about voice search.

Further, if a voice interface is used to deliver search results, a search engine should make an audio disclosure that is of an adequate volume and cadence for ordinary listeners to hear and comprehend it.

So, the consumer searches by voice, and if Siri or the Google Android voice is used to give you the search results, then a voice needs to disclose that it is a paid ad if that is part of the search results. It has to be loud enough so people can hear it.

By the way–congrats goes out my friend Aaron Wall, from SEO Book, who got a well-deserved link from the FTC letter!

Want to read more? Danny over at Search Engine Land has more.

Google Webmaster Tools Now Emailing Website Owners Directly

Google Webmaster Tools has changed their policy, and is now going to start emailing web site owners directly if they detect critical or important issues with your web site. Before now, Google would only notify you about these issues via Google Webmaster Tools’ interface–and you had to log in to receive the messages.

Here is the email that Google Webmaster Tools sent out:

From: wmt-noreply@google.com [mailto:wmt-noreply@google.com]
Sent: Tuesday, September 11, 2012 7:42 PM
To: Bill Hartzer
Subject: Email notifications from Google Webmaster Tools

Dear Webmaster Tools user,
We know that having a healthy and well-performing website is important to you. To keep you informed about your site’s status Webmaster Tools will now automatically send you a notification email if there are critical or important issues that we detect with your site.
We’ll only send you email for issues that we think have significant impact on your site’s health or search performance which have clear actions that you can take to address. For example, we’ll email you if we detect malware on your site or see a significant increase in errors while crawling your site.
For most sites these kinds of issues will occur rarely. If your site does happen to have an issue we cap the number of emails we send over a certain period of time to avoid flooding your inbox.
If you don’t want to receive any email from Webmaster Tools you can change your email delivery preferences at https://www.google.com/webmasters/tools/preferences.
We hope that you find this change a useful way to stay up-to-date on critical and important issues regarding your site’s health.
If you’d like more information about Webmaster Tools and building Google-friendly websites please visit http://www.google.com/webmasters.
The Webmaster Tools Team
________________________________________
© 2012 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
You have received this mandatory email service announcement to update you about important changes to your Google Webmaster Tools account.

I have to admit that I have quite a few web sites that I am a verified owner of (mainly client web sites, but tons of other sites I own and manage), and it will certainly be interesting to start receiving these messages via email. It will definitely save me some time, although I usually log into Google Webmaster Tools daily anyway.

I applaud Google for making this change–it will definitely help some web site owners who don’t make it a habit of logging into Google Webmaster Tools on a regular basis. I’m just surprised that it has taken this long in order for Google to start doing this.

Google Underlining Typos in Search Query Field

Google is always testing part of their interface, and I believe that this new test will most likely be one that sticks around. Google has started underlining typos in the search query field. So, when you start typing a keyword, if what you are typing is an obvious typo, then Google will underline it in red, giving you a chance to correct it.

Here are a few screen captures of what this looks like:

In this case, they keyword searched is the beginning of the word “hello” with several letter “L”s in the search field.

What is interesting is that as of the writing of this blog post, you can see this in Canada only. I tried using http://www.google.ca/ and can see it just fine. But so far I am not seeing this functionality in Google.com. Maybe it’s because Canadians have more trouble typing? (I’m joking!)

A hat tip goes out to JS Lascary for this tip.

Update:Barry pointed out to me that this was tested back in October but it now appears that it’s being rolled out to more people now.

Google Sidewiki Closing – How to Export Your Entries

Chalk this up on the list of another one of Google’s products that failed. Well, I wouldn’t call Google Sidewiki a total failure, but rather a Google product that just did not “catch on” with everyone. Google Sidewiki is being closed down, and Google is offering us all a chance to export all of our entries.

To export your Sidewiki entries, go to http://www.google.com/sidewiki/ and follow the instructions. If you wish to export your Sidewiki entries, you will need to do this before December 5, 2011, after which time Google will be discontinuing support for Sidewiki and deleting all content.

Google is sending out emails to all of the Google Sidewiki account holders, reminding us of the fact that it’s being shut down. Here’s a copy of the email I received today:

Personally, I believe that it didn’t catch on with everyone simply because of the fact that it was so difficult to even get to Google Sidewiki. Not only did you have to have the Google Toolbar installed, you had to make sure that it was turned on. Sure, it was an interesting product at first–and could have caught on if it was not so darn tough for the typical Google users to implement and use. One thing that they could have done was to implement Google Sidewiki as part of the organic search results. Perhaps rather then making it so difficult to install and get to, Google Sidewiki could have been implemented as “reviews” are implemented for local businesses.

In any event, Google Sidewiki is being closed down. Here are the instructions for exporting your entries:

How to export your Sidewiki contributions

After December 5, 2011 Sidewiki will be closed down and all content will be deleted. To keep your Sidewiki contributions you must export them before December 5, 2011. You can do this by following these instructions:

1. Sign in to your Google account
2. Go to All my entries to see all your Sidewiki content created by you
3. Save the page on your computer and check that the saved page is readable

So many people think of Google as being a powerhouse on the Internet. And we sometimes forget that Google isn’t always the incredible company that we think that they are–Google has product failures. And Google Sidewiki is one of them.

Search Engine Rankings for September 2011

Every month about this time comScore releases the search engine rankings data, and I report it here. These are not the actual search engine rankings that you see when you search at a search engine, which are constantly changing.

comscore search engine rankings

comScore, Inc. (NASDAQ: SCOR) , a leader in measuring the digital world, has recently released its monthly comScore qSearch analysis of the U.S. search marketplace. Google’s web sites led the search market in September 2011 with 65.3 percent of search queries conducted in the United States.

More than 17.1 billion searches were conducted in September 2011, and Google Sites ranked first with 11.2 billion search queries (this is up 1 percent over the previous month).

Yahoo! came in second with 2.6 billion search queries during the month of September 2011, followed by Microsoft with 2.5 billion search queries during the month. Ask Network delivered 507 million searches, followed by AOL, Inc. with 265 million (up 16 percent). I am actually pretty surprised by the number of search queries on AOL, and that it has risen 16 percent from August 2011 to September 2011.

Also notable was the fact that in September, 67.4 percent of searches carried organic search results from Google (vs. 66.8 percent in August) while 26.7 percent of searches were powered by Bing (vs. 27.1 percent in August).

Search Engines Not Doing Enough to Stop Hackers

The search engines, Google, Yahoo!, and Bing are great at helping us find information on the web. If you are like me, I personally perform a lot of search queries every day, finding new information and visiting new websites every day. That’s why I use a search engine. However, the search engines are indexing too much information on the internet–and as such, they are not doing enough to stop hackers. They are helping them.

According to a new report by Imperva, their research shows that during an attack, hackers can generate more than 80,000 daily search queries to probe the Web for vulnerable Web applications. Hackers continue to use search engines such as Google, Yahoo!, and Bing, to find websites that are vulnerable–and which sites they can hack into and cause problems for website owners.

Let’s take, for example, the search query above. This is a search for “powered by oscommerce” “catalog” which use the OS Commerce application. That search query is not typical. There is really no reason why someone would want to find all of the websites that contain the entire catalog of websites powered by the OS Commerce shopping cart software unless they were up to no good.

There are literally hundreds, if not thousands, of search queries that are not ‘typical’ search queries. The search engines are not doing enough to stop these queries from being made, time and time again. In this latest test, hackers were able to query Google over 80,000 times in one day, pulling data about websites that were vulnerable. If Google had intentions of stopping hackers, search queries such as these would not be able to be performed 80,000 times in one day.

According to Imperva, here is how hackers set up automated queries in order to get information they need from the search engines:

The Hacker’s 4 Steps for an Industrialized Attack:
1. Get a botnet. This is usually done by renting a botnet from a bot farmer who has a global network of compromised computers under his control.
2. Obtain a tool for coordinated, distributed searching. This tool is deployed to the botnet agents and it usually contains a database of dorks.
3. Launch a massive search campaign through the botnet. Our observations show that there is an automated infrastructure to control the distribution of dorks and the examination of the results between botnet parts.
4. Craft a massive attack campaign based on search results. With the list of potentially vulnerable resources, the attacker can create, or use a ready-made, script to craft targeted attack vectors that attempt to exploit vulnerabilities in pages retrieved by the search campaign. Attacks include: infecting web applications, compromising corporate data or stealing sensitive personal information

Google is not doing enough to stop hackers. Imperva, in their report, explains. “Although Google Hacking has been around – in name – for some time, some new innovations by hackers require another, closer look. Specifically, Google, and other search engines, put in place anti-automation measures to stop hackers from search abuse. However, by using distributed bots, hackers take advantage of bot’s dispersed nature, giving search engines the impression that individuals are performing a routine search. The reality? Hackers are conducting cyber reconnaissance on a massive scale. ”

Imperva’s report, titled “Hacker Intelligence Summary Report – The Convergence of Google and Bots: Searching for Security Vulnerabilities using Automated Botnets“, is a fascinating read.