Did you find this blog post through Bing? Probably not—67% of worldwide searches go through Google, 18% through Bing. But Microsoft has advertised in a substantial TV campaign that -- in the cyber analog to blind taste testing -- people prefer Bing “nearly 2:1.” A year ago, when I first saw these ads, the 2-1 claim seemed implausible. I would have thought the search results of these competitors would be largely identical, and that it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.
When I looked into the claim a bit more, I was slightly annoyed to learn that the “nearly 2:1” claim is based on a study of just 1,000 participants. To be sure, I’ve often published studies with similarly small datasets, but it’s a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I’m guessing is a low six-figure study.
To make matters worse, Microsoft has refused to release the results of its comparison website, Bingiton.com. More than 5 million people have taken the Bing-It-On challenge – which is the cyber analog to a blind taste test. You enter in a search term and the Bing It On site return two panels with de-identified Bing and Google results (randomly placed on the right or left side of the screen). You tell the site which side’s results you prefer and after 5 searches the site reveals whether you prefer Bing or Google. (See Below)
Microsoft’s soft ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000.