Experiments in Business

Here’s an article in today’s Financial Times about a class on business experimentation that John List and I taught at the Booth School of Business. It does a nice job of laying out our philosophy regarding data and experiments.

Thankfully, the reporter did not mention that most of the students hated the class.

Michael Inadomi

This reminds me of some experimentation computer game company Valve has done: through a series of weekend sales on their electronic distribution platform "Steam" (where the marginal cost of selling a game is probably pretty low), Valve discovered that most people sell computer games for too much money, and demand for the games is very elastic. For example, a 10% discount saw a 35% increase in sales revenue, a 25% discount saw a 245% increase in sales revenue, and a 75% discount led to a whopping 1,470% increase in sales revenue.

Shiraz Malik

There were many of us who really enjoyed the class - in fact, having a non-profit background, I found it to be one of the most useful classes that I've had so far at Booth.

The importance of running experiments that are well-thought out and designed for maximum utility cannot be overstated. I'd recommend this class to any MBA students for next year (provided you guys are still teaching it!)

Keith Gillette

I hope the answer is yes, and in the Evening program. I have "Using Experiments in Firms" on my Booth course plan wish-list!


wow.. it's great inspiration.. thanks..
I'm really love this words:
"With the right data, you can improve your decisions,"


Samad Ramoly

Teaching MBAwallahs how to avoid outsourcing their thinking is indeed a laudable initiative. As a matter of fact, it ought to be the basis of formal education itself.


To my mind, a course of this nature would benefit any student for the reason that in the purportedly evidence-driven work environments, all professionals should be able to take apart the data and prescriptions provided by consultants. Many professionals and especially business people are overawed whenever consultants make prescriptions based on some data or experiment. I consider natural experiments very interesting because what they state would probably be more real and the results less likely to be contrived. However, i think that spotting natural experiments is probably the most difficult thing. That some students did not like the course while other did is itself some natural experiment.

XP Student

Did you teach this in the full time or evening program? I'm an Excecutive Program student and think that this class might be a better fit for the XP program.

The average years of experience in the XP program is much higher than either the evening or full time program so most of us have experienced what you describe and would find a more rigorous methodology for dealing with it useful.


I am a mgmt consultant and completely agree that companies need to launch more "experiments" to test hypotheses rather than spend time trying to overanalyze their existing data (which is typically really bad) to derive false insights.

Sounds like a fascinating class, hope you continue to offer it! (Class of 2012 hopeful)


Being an ex-Booth student, freakonomics fan and follower and data-analysis professional in an airline I would very much appreciate to access the course documentation if it where possible as it is with MIT courses.

science minded

When a business other than science conducts experiments, I am wary of the intent---for there is knowledge and the use to which it is put. Leave the experimenting to scientists. am not against the use of scientific knowledge by business-- but have found that when the two are joined- the research suffers. the agenda (policy) of the institution paying for the research becomes the dominant perspective guiding the research. Hence, nothing is really learned. Busniness like to convince themselves of the fact that their policies are working even when they are not.


I've an MBA, and now teach MBAs - here's my 2 cents.

Most of our students frankly don't have a comparative advantage in statistics and struggle with hypothesis testing even without the added complications of causality, selection biases etc. (This may be why Dr Levitt's students didn't like the course!)

Maybe what we should be teaching is upstream? E.g. how to recognize a business problem which needs experimentation? I believe there's still a well-paying market for "business analyst" skills, necessary to structure the problem for downstream execution by partners.

Jonathan Marek

Fascinating article! The company where I work does exactly this for a living -- working with retailers, financial services, and consumer products companies to design and evaluate business experiments. There is a lot more demand for Test & Learn among large companies than there was only 5 years ago, so I'm glad there are b-school courses emerging in this area. Believe me -- it can help your students to get a job and to help their employers make better decisions!

As for the comments from "science minded", I think that is a valid concern, but one that has to be overcome. As with all good business decision making, testing (and analytics generally) require that the practioners are ruthlessly honest with themselves in order to get beneficial results. That ruthless honesty is something we ought to be encouraging in the business world.

Keith Gillette

Dan Ariely, Professor of Behavioral Economics at Duke University and a visiting professor at MIT's Media Laboratory, argues for exactly this type of business education course today in MIT's Technology Review blog: