Could it be that Google refers less because questions are often answered in Google itself?

I don't know about the typical Google searcher, but when I'm looking to define a word, get a quick stock quote, figure out the spelling of someone's name, or remember the date of a historical event, I don't usually have to click a result to get an answer -- it's in a onebox or in the snippet.

Are Google's lower referral numbers evidence of better relevance?

• Posted by: mb
• September 28, 2007 5:48 PM

I doubt it, mb -- especially for knowledgable users, there are plenty of more foculased, far more effective engines available. Google, Yahoo and MSN are more for the "one-size, fits-all" (novice) users.

For example, http://onelook.com is a quite effective meta-engine that searches across many "reference" properties such as dictionary.com and wikipedia.org (and many dozens of others, too). So if you want to find a good answer to your question rather than simply a "quick and dirty" answer, you might use such a site as that.

I also expect that the "conversion rate" at such sites is far higher. A smart advertiser should be interested in co-branding with such "preferred" properties -- but then again there is the issue of whether these sites remain "independent" or not. Certainly, if they are taken over by a big media company, then their reliability would be far more questionable.

Thousands of such sites exist, but few rank well on some of the major search engines (perhaps due to competition -- much like Google's VP of Engineering said at last year's "Press Day" that Google did not want to see http://creditcards.com ranking highly for a search on "credit cards").

Most knowledgable users recognize that Google results are filled with spam. I hardly use "one-size, fits-all" search engines at all any more for precisely this reason. In this light, Jeff Weiner's remarks during "Reinventing the Audience" session at the Conversational Marketing summit (see http://federatedmedia.net/events/index ) rang quite true: Yahoo has more or less been "all about community" all along -- whether that was the portal audience looking for expert guidance in the early years, or whether it's the newer, more "collaborative" communities like flickr.com and/or del.icio.us: it's very much in the "social" space (versus Google, which used to be algorithmic [before the "miserable failure" censorship took over]).

The other "one-size, fits-all" engines? I really cannot say -- they still seem to be "developing" quite rapidly.

What about focused engines such as http://hotels.com , http://pizza.de or http://cellreception.com ? Do you suppose the the "search fulfilltment" rate is higher here? I would think so -- but that's just a guess (and much like the above example [ http://onelook.com ], I would expect that here the "experienced" user will be able to separate the wheat from the chaffe [so I would expect that REPUTABLE advertisers would steer clear of spamming techniques and instead cough up and pay for co-branding instead]).

• Posted by: nmw
• September 28, 2007 9:16 PM

John, by your logic Live search is the best! Thanks for giving a logic to us to justify ourselves as the best:)

• Posted by: Kamal Jain
• September 29, 2007 12:56 PM

I agree with John - you cannot conclude anything just from looking at these numbers. In order to say anything, one would have to have access to some of the queries and see which ones resulted in referral, which ones were reformulated, which ones look like queries that may have been answered directly by the engine, say through the result snippets. Of course, as an academic at a University I realize that I will never get access to such data. mb's guess is an interesting one. It might also be that google users first try simpler queries and then reformulate, while yahoo users immediately type in the longer query. Lot's of possibilities there.

I strongly disagree with nmw, however. Many advanced searchers use standard engines such as google. I use it and am not exactly naive about search. Whether you use a general engine or some other tool is not so much a matter of novice or advanced, but of what you are looking for. For my most common needs (politics, computer science, general trivia, and many navigational searches) a general engine works just fine.

• Posted by: TS
• September 29, 2007 2:45 PM

Yes, TS: I do agree that most "general" engines (as you call them) usually work fine for "navigational" searches (basically: looking up a name like "TS" in the WHITE PAGES). I would be very careful about looking up topics on either politics or computer science with such a "general" ("one-size, fits-all") search engine. As I have observed above: CENSORSHIP HAPPENS. And when I want to learn something (e.g. about computers and/or computer science) I will usually go chat in one of the geeky communities I belong to and thereby recieve far more useful (or whatever you want to call it) results than a "general" SERP which "runneth over" with spam.

Don't know much about trivia -- but I do know what a wonderful web this could be....

;D nmw

• Posted by: nmw
• September 29, 2007 11:51 PM

nmw - I think there are two questions here: (1) do advanced users primarily use general engines or other tools, and in particular do enough of them use other tools for it to make a difference in those stats that were presented, and (2) whether they should (which doesn't directly relate to those charts).

For (1) I suspect that the answer is negative, and that the users that do use other tools are a fairly small group, probably overrepresented amount searchblog readers. In particular, your first statement seemed to say that advanced users don't use the basic engines anyway and thus should be discounted as an explanation for the differences. I doubt this, and think many advanced users use general engines. Thus, there may be significant differences between the general engines in terms of user sophistication - in particular the ways in which MSFT acquires traffic to their engine may significantly impact the types of queries they get.

For (2), I don't argue against using other tools. But I think even for a topics such as politics that are subject to a lot of manipulation, you can get good results via general engines. It all depends on how you interact with the engine and its results, and what questions you ask, and on your personal habits and preferences which are often not rational. (Do you strongly prefer a single interface or are you willing to go to different places for different things? Do you prefer to type or to look over more results? etc.) I can also see that for certain occupations, say in the context of media or law, other tools may be really preferable, but for many other cases it is not so clear.

Personally, I prefer simply dumping a single word or two into google, hitting return, and then going from there. Key to this working for me are the sub-second response times (think of it as hitting return quickly before typing the second word, instead of waiting until the end of the query) and the spelling correction (I type queries fast). By the time other have reached their specialized engine, I have my first result page. And since I know a little about how engines work internally, I can often guess what term combinations might work. But a lot of this is just habits, and few people study their habits to discover possible efficiency gains, unless it is about something they do a lot. So maybe I would do better by changing my habits.

• Posted by: TS
• September 30, 2007 3:35 PM

I should add that by computer science I meant mostly the academic side of the field, which is not really spammed that much. For general technology issues (product reviews, programming etc) things are different of course.

• Posted by: TS
• September 30, 2007 3:43 PM

Well, as you said in your first response, our answers to this issue are probably at best "educated guesses". I should add that I am quite amazed that you feel typing in 1 or 2 words into Google might get you a result that was worth moving your fingers for (unless the strings are something like "ebay" or "yahoo com" -- BTW: I just searched for "delicious" and the top 3 results had nothing to do with "food that tastes good", they were entirely about the site del.icio.us; this further substantiates the idea that Google is primarily useful for people who still have difficulties navigating to sites they wish to visit).

My work (in "information retrieval") is very interdiscipliary in nature. I talk with professionals and academics from many fields/disciplines about their "search behavior" (and/or "information gathering") quite often. Few mention Google. I would say that "colleagues" are a primary source, but also professional journals and/or conferences or even just *random people on the street*. Especially people who have had the experience of "Garbage In, Garbage Out" WRT trying to find reasonable answers to questions by using "dirty data" would (I guess) be prone to be skeptical about Google's rather simplistic algorithms.

I mentioned above that "censorship happens". Well, of course it does! Most people do not want to see a 9 year-old kid walking out of the children's library with a copy of Hustler magazine. But beyond such obvious examples, it is also quite reasonable for people to apply one measuring stick in one instance (like mb's example of "finding a definition") and another measuring stick in another instance "finding a telephone number") -- especially if they are focusing on one discipline of knowledge and hence using that discipline's jargon and not the "GENERAL" language of the wider community. So when a computer scientist refers to "memory" then that would most likely have nothing to do with what a psychologist might refer to using the same term. Therefore, traditional "finding aids" (in the professional and/or academic arenas the primary example of such tools are often referred to as "indexing services") usually focus on particular topics of interest to the "information searcher" and quite often also specify an "authorized" indexing vocabulary (with short definitions or "scope notes").

Not only does Google do nothing of the sort, but spammers actually pretend to portray apples as oranges (and vice versa). The chance of getting a relevant result for non-navigational searches are quite dim (I guess that answering a "trivia" question such as "what is Benjamin Franklin's birthday" is to some degree also a navigational question, insofar as it is primarily a matter of navigating to the entry for "Benjamin Franklin" -- but even here I suspect that historians and/or genealogists would tend to use more specialized engines).

In the early days Google was useful because the Internet was filled with academics (and the method of using citations actually has a very long tradition in academia -- though here too "citation analysis" produces only "ballpark" figures [at best]). The Internet is no longer as "academic" as it was when Google started out -- and that is perhaps one primary reason why it no longer works as well as it did back then. And perhaps the reason it has remained popular in the meantime is that the teenagers who grew up with it had virtually nothing to compare it with. Now that's changing -- and I would say that enabling each person to choose their own search and/or ranking algorithm is a good thing.

And I feel that is another point where we do agree after all: You asked "Do you strongly prefer a single interface or are you willing to go to different places for different things?" I think children's library exist for one reason, pornoshops exist for other reasons, and research libraries exist for yet other reasons. And yes, I do feel that that is a good thing. That the video cited in http://www.telegraph.co.uk/news/main.jhtml?xml=/news/2007/09/28/wburma528.xml was yesterday the second-most viewed video on youtube.com is a little disconcerting. I find that the video shows EXTREMELY disturbing images -- vile and repugnant images that I do not wish tender young children to be exposed to (at least not without *some* parental guidance).

• Posted by: nmw
• October 1, 2007 1:30 AM

I'm with mb. Rather than saying "Yahoo fulfills the most" I'd say "Google already answers the query in its snippet results". I also use Google a lot for spelling checks, telephone number searches which with the right query 9/10 times show in the snippets right away. Even when I need a definition the snippets it picks from say Wikipedia are often enough to get the gist.

There's also the site: link: and other commands in Google which normally don't lead to a click through. Yahoo has Site Explorer for this but I doubt they took that subdomain into equation for this research.

Image search is another area I rarely click through and I asusme here Google Images is bigger than Yahoo's.

• Posted by: Johan
• October 1, 2007 1:49 AM

Fascinating! I do, however, feel that there are people who actually type "delicious" into Google rather than into the location bar -- simply because they do not want to type in the dots.

But apart from that, we have:

1. telephone numbers

2. spelling check, definitions (hmm -- does mb mean the definitions provided by http://answers.com ?)

3. trivial pursuit

4. stock quotes

So including "navigational search" (i.e. the person prefers to type "ebay com" into Google rather than typing "ebay.com" into the location bar), that's 5 ways people have suggested that Google's results are useful to them. Are there any OTHER ways in which people have found Google to be useful?

• Posted by: nmw
• October 1, 2007 3:16 AM

Very interesting discussion. I have to admit that many of my queries fall into the above 5 categories. I never type into the address bar because it is a pain - positioning the cursor, doing the dots, the http:, etc. And many of my computer science queries are either navigational, or very specific (not "information retrieval" but with much more specialized terms and often names), and I find google to work well in that case. Often I just use google as an entry point into citeseer, where I browse for literature, but I rarely search within citeseer. And quite a lot of my (non-CS) questions are answered by the result page.

While these 5 categories may not seem very challenging for an IR system, I would suggest that these 5, and other types of queries between these and "traditional informational queries", are the drivers in terms of what makes an engine successful and in terms of how search impacts technology at large. You can't really separate these from "real search" -- if you try then for most people there will be very little left. And the boundary between navigational and informational is not always that clear - so do not disrespect the humble navigational query :)

• Posted by: TS
• October 1, 2007 6:18 PM

I have great respect for the navigational query -- the question is: Does Google have a similar respect for it? AFAIK, Google employs a large army of lawyers in legal battles concerning how Google "sells" results for trademarked strings to the highest bidder (often to competitors of the trademark owners). And AFAIK the "snippets" also may someone else's intellectual property (i.e. protected by copyright). So for most of the 5 examples described above, Google is actually in "deep doodoo".

• Posted by: nmw
• October 2, 2007 12:18 AM

The chance of getting a relevant result for non-navigational searches are quite dim

I tend to agree with this statement. Google is being consciously optimized for navigational searches, at the expense of other types of searches, such as explorational, informational, transactional, etc.

I suppose that's ok, on the surface of things. It certainly is the "lowest common denominator" approach to information retrieval, something that will satisfy the majority of the crowd, the majority of the time. So it is completely understandable why they optimize in this direction.

The danger, in my mind, is when we take this type of information retrieval and equate it with all of information retrieval. When we stop looking for something better, something more, that is not good. When we fall into the habit of using Google a certain way, because we "kinda know how search engines work" and know that we can get millisecond response times, we've basically fallen into a local minimum. If we never even try to get out of that minimum, it might as well be game over. If customers are not demanding more of the search engine, the search engine is never going to bother making any real improvements beyond constant spam monitoring. The search engine will forever remain in navigational mode.

Is that what we want?

• Posted by: JG
• October 2, 2007 7:17 AM

A new search tool called PredictAd is offering a service that offers individual websites and blogs similar features to what the big guys like Google, Yahoo and MSN have added to their search engines (autocomplete, rss live search).

Check out this article on PredictAd to get an idea of what it does:

http://www.ringofblogs.com/2007/10/01/predictad-navigation-monetization-and-search/

• Posted by: tomer molovinsky
• October 2, 2007 9:35 AM

In a previous job, my team performed a great deal of analysis into the quality of search engine results, as measured by user response to a results page. In one project, we took four leading search engines and transformed their UIs into wireframes (stripped out all the formatting and branding elements). Then we asked users to perform identical searches on each of the four engines and evaluate the quality of results. In this brand-neutral test, users consistently reported that one or more engines outperformed Google. Yet, when we revealed the brands behind each wireframe, existing Google users maintained their preference for Google. Never underestimate the power of a brand.

My team also looked at search referrals as a measure of user satisfaction and, as with the Compete study, found that users more frequently found a "clickable" result on engines other than Google.

The findings of these research studies held up for each of the three types of searches: navigational, informational, and commercial. And let's not flatter ourselves by thinking that Google users are more sophisticated or advanced, and thus need only to read the text string in the search results to find out what they need to know. There's no evidence for this.

Put simply, research evidence consistently suggests that the leading search engines, incuding Yahoo, Ask, and some metasearch engines, produce user satisfaction levels that match or exceed those of Google users.

• Posted by: p'ang
• October 2, 2007 11:38 AM

"we can get millisecond response times"

Yes: *time is money* (why does it always seem like Benjamin Franklin was 10 times smarter than anybody else? ;)

Are Mozilla and IE all there is -- I mean: is that all there is?!? (no there's another one isn't there? what was that one called again? -- oh, never mind...)

Has anyone ever used ctrl-L ? OK, that's one millisecond.

"[Google] will forever remain in navigational mode"

I'm not even sure that Google will forever remain a search engine (right now it seems to be more of an ad agency). What's really surprising is that there is this MYTH that Google is a search engine (to be reckoned with ;) -- and people continue to believe it! Typing "ebay" into Google is actually a WASTE of time -- so if "time is money", then it's *time to move on*....

Another amazing thing is that it seems like there is a yearning for some HERO and/or ROLE MODEL. If I go to http://www.christianity.com then I can EITHER search the site or the Bible (that's kinda neat-o! :) And if I go to the blog section of the site, I can do a keyword search (I tried "help", then I tried "support": first result was "how to handle a money crisis" -- not so bad! ;) I am not sure that all of my problems need to be solved from a particularly Christian.COM perspective -- what other measuring sticks are there? (again: is that all there is?!?)

How about http://help.com? I can search for tags: first "money", then the site offers further tags to hone in on -- I click on "loans"... -- this is also neat-o! ;D I type in "save money" (minus quotes) and then I see something new: I can limit my search to "posts" or "replies" or "both" (COOL!! ;) -- I can add user names, ... -- is this a search engine? Is this advanced? Is this explorational? informative? ...?

When I type "help" into Google, help.com shows up as the first result. Is that good or bad? What is the most efficient / effective method an advanced searcher should use?

• Posted by: nmw
• October 3, 2007 9:36 AM

One way to solve this mystery might be to by using the process of elimination -- which methods are NOT efficient / effective?

Here's a start:

http://groups.google.com/group/blogger-help/search?q=find+blog

• Posted by: nmw
• October 4, 2007 1:45 AM