Is the "Bing It On" Challenge Lightweight?

Microsoft has now responded, with a blog post and a letter, to my post about an experimental study that I coauthored with Yale Law School students Emad Atiq, Sheng Li, Michelle Lu, Christine Tsang, and Tom Maher.  Our paper calls into question the validity of claims that people prefer Bing nearly two to one.

In response to several commenters: I do not work for and do not have any consulting relationship with Google.

Microsoft claims that our study is flawed because it relied on their own blind comparison website.  They now say that "Bing It On" is meant to be a "lightweight way to challenge people’s assumptions about which search engine actually provides the best results."  To be sure, companies often use fantastical or humorous scenarios for free advertising. However, Microsoft’s television commercials present the site as a credible way that people can learn whether they prefer Google or Bing.  These commercials show people who discover that they really prefer Bing to Google.  The challenge site that they created is either sufficient to provide insights into consumer preferences or it isn’t.  The advertisements give the impression that the challenge site is a useful tool.  Microsoft can’t have it both ways.  If it is a sufficient tool to “challenge people’s assumptions,” then it is sufficient to provide some evidence about whether the assumed preference for Google is accurate. 

Challenging the Bing-It-On Challenge

Did you find this blog post through Bing?  Probably not—67% of worldwide searches go through Google, 18% through Bing.  But Microsoft has advertised in a substantial TV campaign that -- in the cyber analog to blind taste testing -- people prefer Bing “nearly 2:1.”  A year ago, when I first saw these ads,  the 2-1 claim seemed implausible.  I would have thought the search results of these competitors would be largely identical, and that it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.

When I looked into the claim a bit more, I was slightly annoyed to learn that the “nearly 2:1” claim is based on a study of just 1,000 participants.  To be sure, I’ve often published studies with similarly small datasets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low six-figure study. 

To make matters worse, Microsoft has refused to release the results of its comparison website,  More than 5 million people have taken the Bing-It-On challenge – which is the cyber analog to a blind taste test.  You enter in a search term and the Bing It On site return two panels with de-identified Bing and Google results (randomly placed on the right or left side of the screen).  You tell the site which side’s results you prefer and after 5 searches the site reveals whether you prefer Bing or Google. (See Below)

Microsoft’s soft ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000.

What Does Your Web Browser Say About Your I.Q.? (Hint: I.E. Users Won't Like the Answer)

See ADDENDUM (8-3-11; 9:13am EDT) below

A study by AptiQuant Psychometric Consulting finds that people who use Internet Explorer as their web browser are, on average, less smart than those who use other browers. As PC Mag reports:

Over a period of around four weeks, the company gave a Wechsler Adult Intelligence Scale (WAIS) to users looking for free online IQ assessment tests, then recorded the results and browsers used for all participants above the age of 16.

Across the board, the average IQ scores presented for users of Internet Explorer versions 6 through 9 were all lower than the IQ scores recorded for Firefox, Chrome, Safari, Camino, and Opera users.

Food and the New Physics

Molecular gastronomists are altering and reimagining our food: from flavored foams to dry ice for dessert. But you have to wonder, have the practical applications of science in the kitchen taken a back seat to all this whimsy?