Challenging the Bing-It-On Challenge

Did you find this blog post through Bing?  Probably not — 67% of worldwide searches go through Google, 18% through Bing.  But Microsoft has advertised in a substantial TV campaign that — in the cyber analog to blind taste-testing — people prefer Bing “nearly 2:1.”  A year ago, when I first saw these ads,  the 2-1 claim seemed implausible.  I would have thought the search results of these competitors would be largely identical, and that it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.

When I looked into the claim a bit more, I was slightly annoyed to learn that the “nearly 2:1” claim is based on a study of just 1,000 participants.  To be sure, I’ve often published studies with similarly small data sets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low-six-figure study. 

To make matters worse, Microsoft has refused to release the results of its comparison website, BingItOn.com.  More than 5 million people have taken the Bing-It-On challenge – which is the cyber analog to a blind taste test.  You enter in a search term and the Bing-It-On site return two panels with de-identified Bing and Google results (randomly placed on the right or left side of the screen).  You tell the site which side’s results you prefer and after five searches the site reveals whether you prefer Bing or Google (see below).

Microsoft’s ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000.

 bing

So together with four Yale Law students, I set up a similar-sized experiment using Microsoft’s own BingItOn.com site to see which search engine users prefer.  We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were “ties”).  This is not even close to the advertised claim that people prefer Bing “nearly two-to-one.”  It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge, if, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.  Microsoft might have realized this and has more recently altered its advertising to back off their original claim to just say that people “prefer” Bing.

We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred.  We randomly assigned participants to search for one of three kinds of keywords: Bing’s suggested search terms, popular search terms, and self-suggested search terms.  When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google).  But when the subjects in the study suggested their own searches or used the web’s most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing.  These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results. You can read our full paper here.

The upshot: several of Microsoft’s claims are a little fishy.  Or to put the conclusion more formally, we think that Google has a colorable deceptive advertising claim against Microsoft.  It could be worth a lot of money on lost ad revenue if the claims misled people into thinking that a substantial majority of people prefer Bing over Google.  Then again, I might be a little over zealous in seeing Lanham Act violations.  Back in 2010, I suggested that the movie Date Night might contain a deceptive Kindle ad.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 68

View All Comments »
  1. Brendan says:

    I think it’s telling that I used google to search for the bingiton challenge.

    Well-loved. Like or Dislike: Thumb up 78 Thumb down 6
  2. Bodger says:

    A little-known fact but one which the Microsoft folks clearly didn’t research (Google?) before picking a name for their search: in northern-English dialect, ‘bing’ is the accepted name for a waste heap, some of which can the scale of small mountains.

    Well-loved. Like or Dislike: Thumb up 35 Thumb down 13
    • j fre says:

      Hidden due to low comment rating. Click here to see.

      Disliked! Like or Dislike: Thumb up 27 Thumb down 42
    • Viking says:

      At least where I live, in Norway (in my dialect of the norwegian language); To be “completely Bing” or to be “completely Bing in your head” has been used (as long as I can remember) of people that’s stupid or being too bright for their own good…

      I have preferred Google for web searches since the time everyone else preferred Alta Vista. When Bing appeared, it somewhat became the default in my browser, and I got really angry because I never found what I was searching for without browsing through several pages of irrelevant listings. It was actually long after I got rid of the dreadful Bing search that I found out that Microsoft was behind it.
      My first thought then was; Why would a world leading company select such a stupid name for their search engine, when they obviousely have unlimited funds and resources available to find the best available name for their product..?

      Thumb up 5 Thumb down 1
      • Reno says:

        I got really pissed at it to kinda for the same reason. I typed in yugioh-card.com and it, for 5 pages, gave me nothing but hentai sites. I was 12 and my mom smacked my and said what the f@$& are you looking at. I prefer google because its not as reverted

        Thumb up 0 Thumb down 0
  3. Stuferus says:

    i prefer duckduckgo at the moment!

    Well-loved. Like or Dislike: Thumb up 25 Thumb down 11
  4. Mskidee says:

    I have wondered if this claim was misleading as Google still seemed to be the browser everywhere. I took the Bing It On challenge and chose Google 5 out of 5 times. In one of the searches I used the name of a relatively young non-profit in NY called It Could Happen To You Inc. In the Google results the top results linked to the organizations website (although I did not get the general website link http://www.itcouldhappen2you.org, I did get content pages on from the website). In Bing the closest I got to the organization was its listing on a non-profit listing site. Even when I used a celebrity name as my search Google showed the official site first while Bing showed wiki and photos. The commercials are very misleading and I wonder if this meets the level of false advertising which may be why Microsoft refuses to release their findings.

    Well-loved. Like or Dislike: Thumb up 13 Thumb down 7
  5. Sandra says:

    I can’t even do the experiment because I’m in Mexico and the “bingiton” page send me to Bing Mexico, and limit all my searches to Spanish.
    I hate Bing and I always will.

    Well-loved. Like or Dislike: Thumb up 28 Thumb down 13
    • Deja-Q- says:

      You’ll always hate Bing no matter what. Wow, that’s a lot of hate you have going on there for Bing.

      Well-loved. Like or Dislike: Thumb up 14 Thumb down 9
  6. Kristi says:

    I just tried the Bingiton challenge, which said I prefer Bing, and then I used Bing to pull up the exact same search. The pages didn’t even match! For instance, I searched News, which gave me an in-depth news coverage section, which Bing doesn’t have. That was a Google feature.

    Well-loved. Like or Dislike: Thumb up 28 Thumb down 11
    • Martin Beeby (Microsoft) says:

      Bing has a news section. It can be found at the top of the home page, or at bing.com/news or if the news is recent and relevant to your search it may appear along side your search results.

      Thumb up 9 Thumb down 7
    • Plutonar VS says:

      Explanation: both Google and Bing personalize results rankings (by IP address & estimated location as well as cookie). When you try BingItOn you get neutral rankings.

      Thumb up 4 Thumb down 3
  7. James says:

    For myself, it depends on what I’m doing. For a general search on a strange computer, I’d use Bing, simply because it doesn’t have that horrible flickering autocomplete thing. And Bing maps are better than Google’s, at least for finding backcountry dirt roads. But it doesn’t have things like Google Scholar, or (at least that I’ve found) a good advanced search mechanism.

    Thumb up 10 Thumb down 9
  8. Ken Bellows says:

    Another major problem with bingiton.com is that it restricts Google’s results to the central pane of the page. Some of the most interesting results (related products on ebay, amazon, etc., related results from google maps, …) show up in a side pane that is completely cut out of the Bing It On version of the page, while Bing’s fancy extras are included in the same page. This falsely implies that Bing gives more useful results when it doesn’t.

    For an example, try “lawn mower” in bingiton.com and google.com. Then try “Yosemite National Park”.

    Well-loved. Like or Dislike: Thumb up 33 Thumb down 9
    • Rob says:

      Bing has additional result item both in the page but also in the right pane depending on your type of search.
      The BingItOn page strips the side panes from Bing and Google and also the ads that are above the search results.
      But if you search on BingItOn for a celeb for instance you might still see media results (images/video) in the Bing as well as the Google results as those are often shown mixed within the results pane.

      Thumb up 3 Thumb down 3