Mozilla Gets Freaky

For the last few years I’ve been trying to convince businesses to run experiments in order to learn how to do things better. Why is it that experimentation is the gold standard in science, but rarely exploited in corporations? My own hunch is that the main reason is what economists call “path dependence” — in other words, businesses don’t run experiments because they rarely have in the past. If, by chance, Henry Ford‘s innovation had been to run experiments rather than to develop the automated assembly line, experiments would be commonplace today in business.

A former student of mine named Ken Kovash, who now works at Mozilla, provides a nice example of how the simplest of experiments can provide answers that are otherwise elusive. As Ken describes in greater detail on the Mozilla blog, one of the ways Mozilla acquires new customers is through pay-per-click ads on search engines. The question Mozilla had is the following: if someone types “firefox” into a search engine, usually the first result they will see is the Mozilla site, so does it really do Mozilla any good to pay search engines to do featured links? Do ads actually generate more traffic, or do they just shift customers around — e.g., instead of getting the customers free, Mozilla ends up paying the search engine because of the pay-per-click ads? Without performing an experiment of some kind, this is a hard question to answer.

So over a two-week period, Mozilla experimented with turning their pay-per-click ads on and off more or less at random. The findings, as Ken reports, are somewhat mixed. Looking at the data one way, it appears that two-thirds of the customers who normally come to Mozilla through pay-per-click ads would get there anyway. On the other hand, the absolute number of downloads was substantially higher when the paid ads were running. This suggests either that (1) their treatment and control periods were different for an unknown reason; or (2) that the pay-per-click ads lead people to download more often through other channels. My guess is that (1) is more likely to be the explanation. But how can one really be certain?

The answer: by running more experiments.


I think your premise that companies don't run experiments is false. What are focus groups? What is R&D spending? What is price differentiation for a product (based on time, color, style, etc.) I'm also very sure large corporations analyze their advertising spending, which is exactly what was tested in your example.

Aidan Skinner

Corporations do do experiments, but rarely publish the results (because it is rarely in their interests to do so, the results form competitive advantage).

There are exceptions to this, in my own field of software development one of the newer programming methodologies - Agile/Extreme Programming - started at Chrysler and, coincidentally, also spawned wikis.

Sometimes companys themselves *are* experiments, such as Cygnus and RedHat experimenting in commercialising Open Source software.

Not to mention the direct, hard science experiments done by the likes of Biotech companys.

I think a more interesting question is how to get companys to publish the results of these experiments?


Businesses do run experiments, but not always the way you think of in science - concurrent studies with controls.

Businesses don't want to do expensive experiments and they don't want to do experiments that will disrupt a revenue source.

But when a business is failing they are more likely to change things around and experiment with different things.


"[H]ow can one really be certain? The answer: by running more experiments."

Not so. Certainty, in the philosophical sense of the word, is inaccessible to empirical analysis. In reasoning from effects to cause, one cannot in general guarantee that all the relevant data have been accounted for and that the model is correct.

Of course, one may also argue that certainty is unavailable in all realms. But how can we be certain of this?


Okay, the Six-Sigma discipline in business process improvement demands experimentation. In fact, an important part of the toolbox is Design of Experiments, or DOE. Without validation of the improvements predicted by the math, NO six-sigma practitioner worth their belt would recommend taking the improvement to the production floor.

Ian Tindale

Would the frame of assumption be, in performing experiments, that the experimentation will hone efficacy by improvement, or produce new stuff via innovation? Obviously it's likely to do both, but in the minds of those 'allowing' or 'promoting' the experimentation activity (which itself implies a difference between 'buying' and 'selling' the notion of experimentation within an organisation's value tree) is the assumption that we'll no doubt see a benefit, but which way - evolutionarily or revolutionarily?

Nils Davis

Google is famous for doing experiments like this. (So is Yahoo!, so it's not a guarantee.) They have a mantra that they don't make any changes to their home page without test data showing that the change accomplishes some goal. Of course, they have the world's best lab for making experiments like this.

Himanshu Jain

Every decision in a business is an experiment. Because you never really know what will be the output. And if the output is => expectation, it becomes a proven experiment and then repeated to death


Question: Does an experiment help the corporation increase profits for shareholders, as is the legal requirement for publicly traded entities?

And isn't the Visa corporation an extended experiment in chaordic management?

There are of course, axiological differences between corporate and academic experiments. Unless of course, the academic experimenters lose their free inquiry of knowledge and decide upon the corporate model. Then, the axiological differences become moot, as does the value of the research itself.

arnold braff

I spent 18 years running a mail order company. Our company, and I suspect all successful mail order companies, conducted many controlled tests (i.e., experiments) all of the time. Test were so important to our business that our data processing system was designed to facilitate evaluating test --- and, yes, we were quite well aware of sample sizes and statistical significance.

A major focus of our marketing meetings was the development of innovative ideas that were testable. Maybe one in ten new programs was worth continuing but those winners are what kept refreshing the business.

Echoing comments above, companies do experiment, you just don't know it and don't hear the results.

It's inherently easier and cheqper for an internet only company to run experiments such as the Mozilla one mentioned. There are much higher costs when developing tangible physical products, whereas Google can direct you to a test homepage or test resuls page.

A prototype car or washing machine requires pla nt and equipment expenses (holding engineering and design expenses constant).

The Owner's Manual

The art of business consists in making irrevocable decisions based on insufficient information.

If that not experimenting, I don't know what it.

Having spent a career in photofinishing engineering, I can assure you that superior performance is impossible without testing the bounds of what is given.


This isn't really an example of path dependency. In the classic examples of path dependency, a sub-optimal solution is used because that solution was used in the past, and the cost of switching is higher than (or near to) any offsetting gains in efficiency. See, e.g., the QWERTY keyboard. Here, though, the fact that a corporation has not done experiments in the past does not prevent it from doing them in the future - they are not dependent on the path already chosen.

Kevin Friedman

Some of the most profitable companies in their respective industries over the past decade have been religious in running experiments. For example, CapitalOne has dominated their competitors by running experiments through their marketing departments to identify and to sign the most lucrative customers. (note: these are not always the most credit worthy prospects either; in fact, the higher "risk" prospects sometimes made for the most profitable credit card customers.) Also, Harrah's has been incredibly successful since Gary Loveman took the reins in Jan '03 and they have hired a panoply of quantjocks to mastermind their experiments and regressions.

But, simply running experiments and regressions doesn't always mean success. And sometimes just executing on a hunch can work wonders... like when former Gillette CEO Jim Kilts opted to launch the red razor to compete with Schick's Quattro. The marketing minions said a red consumer product is suicide because in the focus groups, the red brought to mind blood, and following, subconsciously trigger the fear of a cut. But the "little red razor" stymied the quattro and preserved Gillette's market share.

In a fast moving market, i'd run with a hunch... but given the benefit of time... testing helps to identify theoretical optimal solutions. Yes, I did get an MBA :)



There are other issues that favor path dependence vs. experimentation in business, especially those that involve substantial capital investment to realize the benefits:

1. It's often far more profitable for the money invested to experiment to improve the yield of an existing process over experimenting to find and implement an altogether new process.

2. Internal company politics. At a lot of companies, the solutions produced by an internal R&D group performing experiments to find new solutions fall into the "Plan B" category.

Here, managers who run existing product lines and who have firmly established their power within the organization have that power threatened by a more powerful manager to achieve certain improvements within their mini-empire, with the threat that the company will pursue Plan B with a different manager running the show instead.

See #1 above for what Plan A (how the manager keeps control over their mini-empire) looks like.

3. A lot of companies have R&D departments that have done some really impressive experiments that are capable of really revolutionizing their industries. A lot of this work is so cutting edge however that no-one at the company knows how to value its potential, especially if its outside the scope of the company's primary product lines. A lot of this work ends up getting shelved and forgotten.

4. The knowledge problem. A lot of the experimentation that happens is done by small, isolated groups within a company. Let's say the results of the experimentation is innovative and might lead to some new profit-making opportunities, but perhaps didn't quite make the company's internal ROI (return-on-investment) requirement for implementing the results of the experimentation for where it was originally intended.

No one else in the company really knows about it, even in areas where it might meet or surpass other ROI targets if implemented, because the company really hasn't invested in developing an effective system for disseminating knowledge within the company. And because the knowledge isn't disseminated, the innovation gets shelved and forgotten.



I think you are incorrect that companies rarely run experiments. The Mozilla experiment you describe is extremely common. It's also common for companies to generate multiple "landing pages" with varying prices, then interpolate and extrapolate supply/demand numbers (with a reasonable assumption of a convex curve) to determine optimal product pricing.

I do think you are sniffing out a real point however. Often time companies don't run experiments when they could, and when the data would clearly help make them better decisions. This only makes sense! All a company can do is guess at the cost of an experiment, and if the company judges that cost to be excessive relative to expected gains (or relative to expected gains volatility adjusted), the experiment does not get run. This is the stuff of good executive mathematics.


I actually have a comment about the experiment's results. Mr. Levitt argues that the treatment and control periods were different, however wouldn't it be logical to assume that the people searching for firefox using a search engine are already aware of it (and thus have it) and are less likely to download it whereas the pay-per-click customers are more probably to never have heard of mozilla or firefox and are therefore more likely to download it. In other words, people who are using a search engine to look for firefox could possibly have technical questions or might seek additional plug-ins, whereas effectively everyone who follows the pay-per-click path is gaining a greater awareness of the existence of firefox?


In reading the comments on this blog I am shocked that so few people understand what an experiment is. "Collecting data" and analyzing such data is not experimentation. Most importantly, focus groups are not experiments. What some comments referred to as "experimenting" by making decisions in novel situations and evaluating the outcomes of those decisions is not research and it is most certainly not experimentation. That is merely a colloquial use of the term.
Dr. Levitt described a controlled experiment, which is usually the most stringent definition of experimentation, but one that is most commonly used by scientists.

Justin James

The biggest problem with business experimentation is that they will never get proof of a causal relationship between the experiment they performed and the results they received. At best, they will get a strong correlation. Unlike economists, business people do not like correlation, they like proof. If your bonus depended upon someone else's perception that your actions definitely caused an increase in profits, you certainly would not go about getting, at best, "correlations", you would be looking for "proof". The only "proof" that the business world really accepts is revenue stream. Why jeopardize the existing revenue stream just for a "correlation"? Why would a bank or venture capitalist fund what is, at best, a "hunch" (also known as "gambling") when they could put their money in a proven business model (this is called "investing").

The more I read the Freakonomics blog, the more I wonder just how in touch with reality the authors are.



michael webster

I agree with Kevin Friedman about Harrah's: but I am not sure that Loveman has run double blind experiments.

He certainly has collected alot of data which allows Harrah to calculate what they think a particular client type is worth.

But I didn't read Loveman as claiming that they ran double blind testing.