Challenging the Bing-It-On Challenge

Did you find this blog post through Bing?  Probably not — 67% of worldwide searches go through Google, 18% through Bing.  But Microsoft has advertised in a substantial TV campaign that — in the cyber analog to blind taste-testing — people prefer Bing “nearly 2:1.”  A year ago, when I first saw these ads,  the 2-1 claim seemed implausible.  I would have thought the search results of these competitors would be largely identical, and that it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.

When I looked into the claim a bit more, I was slightly annoyed to learn that the “nearly 2:1” claim is based on a study of just 1,000 participants.  To be sure, I’ve often published studies with similarly small data sets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low-six-figure study. 

To make matters worse, Microsoft has refused to release the results of its comparison website, BingItOn.com.  More than 5 million people have taken the Bing-It-On challenge – which is the cyber analog to a blind taste test.  You enter in a search term and the Bing-It-On site return two panels with de-identified Bing and Google results (randomly placed on the right or left side of the screen).  You tell the site which side’s results you prefer and after five searches the site reveals whether you prefer Bing or Google (see below).

Microsoft’s ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000.

 bing

So together with four Yale Law students, I set up a similar-sized experiment using Microsoft’s own BingItOn.com site to see which search engine users prefer.  We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”).  This is not even close to the advertised claim that people prefer Bing “nearly two-to-one.”  It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge, if, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.  Microsoft might have realized this and has more recently altered its advertising to back off their original claim to just say that people “prefer” Bing.

We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred.  We randomly assigned participants to search for one of three kinds of keywords: Bing’s suggested search terms, popular search terms, and self-suggested search terms.  When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google).  But when the subjects in the study suggested their own searches or used the web’s most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing.  These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results. You can read our full paper here.

The upshot: several of Microsoft’s claims are a little fishy.  Or to put the conclusion more formally, we think that Google has a colorable deceptive advertising claim against Microsoft.  It could be worth a lot of money on lost ad revenue if the claims misled people into thinking that a substantial majority of people prefer Bing over Google.  Then again, I might be a little over zealous in seeing Lanham Act violations.  Back in 2010, I suggested that the movie Date Night might contain a deceptive Kindle ad.


Karl

This assumes that Bing and Google's search quality have remained relatively constant since the campaign launched. Competition between Bing and Google have pushed both to feverishly focus on improving their search quality.

Of course, this is not always a great thing. Since the campaign launched, Google's algorithms seem to have been tweaked to be more like Bing -- generating better results for common queries at the expense of quality in the "long tail" of uncommon queries.

Clive Portman

Do you think to some extent the reason people like Google's responses to their self-suggested search terms more is because they've been trained to use Google with their everyday use of it?

I've been trying Bing recently, but haven't been finding the results as good as Google. Yet, I've been using Google for years and know exactly how to phrase my Google queries to get the information I'm after.

That learned behaviour is going to be a problem for Bing isn't it? On the surface Bing isn't producing the best results for me, but then I'm searching with Bing using skills appropriate to Google. Why should I have to re-learn the way I search?

Paul T. Lambert

Relearning how do do things is necessary in our fast-changing tech world; otherwise you'll be stuck in the past and merely getting by instead of maximizing what you can do. I always experiment to see if I can do things better and faster with different services.

Unfortunately, websearch is turning into an adversarial process, especially with so many advertisers doing SEO and companies using various secret preferences to include and rank results, perhaps with some "input" from the NSA and other government or private enterprises. There's no such thing as an objective websearch anymore; you have to study results and infer whatever heuristics and random behaviors are in place on each particular day. I suppose any search engine will quickly tell you what Justin Bieber had for lunch or where Honey Boo-Boo just took a dump, but I usually have to search for very specific and atypical items of information. I guess I am not the typical user targeted by Bingoogle, so none of this really matters all that much in the grand scheme of things.

Read more...

Beslin

I need to summarize my thought with a small example, i was working on a website SEO which was in first page first position in Bing about a year but i hardly received 5 inquires. Once my website start listed in Google first page (not in first position) within a month i got more than 50 inquiries.
After that i guessed Google is getting around 90% of search queries and others getting 10%(might be wrong but that is what i believe)

SEO

as an SEO i'd have to say i think google's search results 90% of the time are much better than bings or ddg's. i hate the way google have lumped everything into (not provided) now though, they're a pretty deplorable bunch.

Yardbird

I took the Bingiton challenge and here are two things I found extremely interesting.

1. Google wins hands-down on any non-traditional search terms. Meaning of pryhic (spelled incorrectly on purpose), Time in Tokyo, 19000 INR in USD, Phone number of Indian Bistro Palatine, Chicago Blackhawks. These are not what one would have used search engines for originally and Google gave me the answer I was looking for everytime in the first card it displays, including the score for the last played Blackhawks game. Bing gave me links. On traditional searches, names of people, terms etc both brought up links and I actually preferred Bing links 3-2 in these cases.

2. When doing this, I took the test a few times, it struck me that this is a great way for Bing to collect very valuable data about what people are searching for and why they prefer Google and thus how to improve their own search results. It might not be ingenious but definitely sneaky. :)

Read more...

russ

1000 is the gold standard for polling - there is no reason to doubt that result. The live website results with "de-identified" results are not going to get the same response. Some of the things that are identifiable are the reasons that people have a preference (like # and placement of ads). You are right in that the results are largely the same at both engines. The main thing driving people to bing is that some people are getting a bad feeling (for one reason or another and justifiably or not) about google. Then I would say the second driver is that when you try bing and you see a few of the little niceties they've created you start think twice about ALWAYS using google. The possibility of breaking your google addiction is enticing. At this point I use bing about 99% of the time and I only use google if I need to do a google specific think like a LINK: search.

Mike

Another big part of this: the results from Google that are displayed in the challenge are not even the actual results as it appears on Google.com. They are stripped of the useful, highly visual "cards" that Google is now surfacing for a lot of common searches.

Donatelo

I agree
I do not see that Bing is better
But from another point of view, I have to admit that Google is not better than Bing either. Difference is not that great as it was two years ago, quality of Bing and Google result is more or less the same.

BigD

Bing is a better search engine in my opinion. I stopped using google a long time ago. I still use gmail however the new outlook is 10 times better. Most kids these days use google, its the cool thing to do and one day something else will come along with a catchy name and google will be the myspace of social networks. I believe someone who like bing can make this same experiment lean towards bing.

Klark Kentsky

Google is your friend, but Bing is your bitch.