Mozilla Gets Freaky

For the last few years I’ve been trying to convince businesses to run experiments in order to learn how to do things better. Why is it that experimentation is the gold standard in science, but rarely exploited in corporations? My own hunch is that the main reason is what economists call “path dependence” — in other words, businesses don’t run experiments because they rarely have in the past. If, by chance, Henry Ford‘s innovation had been to run experiments rather than to develop the automated assembly line, experiments would be commonplace today in business.

A former student of mine named Ken Kovash, who now works at Mozilla, provides a nice example of how the simplest of experiments can provide answers that are otherwise elusive. As Ken describes in greater detail on the Mozilla blog, one of the ways Mozilla acquires new customers is through pay-per-click ads on search engines. The question Mozilla had is the following: if someone types “firefox” into a search engine, usually the first result they will see is the Mozilla site, so does it really do Mozilla any good to pay search engines to do featured links? Do ads actually generate more traffic, or do they just shift customers around — e.g., instead of getting the customers free, Mozilla ends up paying the search engine because of the pay-per-click ads? Without performing an experiment of some kind, this is a hard question to answer.

So over a two-week period, Mozilla experimented with turning their pay-per-click ads on and off more or less at random. The findings, as Ken reports, are somewhat mixed. Looking at the data one way, it appears that two-thirds of the customers who normally come to Mozilla through pay-per-click ads would get there anyway. On the other hand, the absolute number of downloads was substantially higher when the paid ads were running. This suggests either that (1) their treatment and control periods were different for an unknown reason; or (2) that the pay-per-click ads lead people to download more often through other channels. My guess is that (1) is more likely to be the explanation. But how can one really be certain?

The answer: by running more experiments.