The Value of Statistics

Last week’s excellent Times article on the high salaries being paid to statisticians is just one sign of the market value of number crunching. As I wrote in the afterword to the paperback version of Super Crunchers:

In 2007, there were three major acquisitions of “business intelligence” firms: in April, Oracle purchased Hyperion Solutions Corp. for about $3.3 billion; in October, SAP purchase Business Objects for $4.8 billion; and, in November, IBM announced its $4.9 billion purchase of Cognos. These multi-billion dollar acquisitions of firms whose sole product is number crunching is powerful evidence that data-driven decision making has market value.

To this list of billion-dollar purchases we can now add IBM’s acquisition of stat software provider SPSS for a mere $1.2 billion.

But in some ways, an even more eye-opening market test is for a much smaller amount. Microsoft shelled out $115 million to buy Farecast.com, a company that crunches numbers to tell you whether you should buy an airline ticket now or wait until closer to when your flight leaves. I’ve long been a fan of the Farecast predictions (which wonderfully include estimates of their own precision):

DESCRIPTION

It was a wonderful decision of Microsoft to use Farecast predictions as a central element of Bing Travel. Still, $115 million surprises me. Farecast has terabytes of data but in some sense it is just running something close to a very simple multivariate regression. It goes to show that thinking up the right regression to run can be worth millions.


Miguel

I agree that the number of staticians and their salaries are growing, at leat by what I have read in your blog and in Tim Harford's blog. But I differ in a way... Those databases have other purposes too, such as run the regular business operations. Their main goal is not to crunch numbers and give out statistics, such as the example of Farecast.com.

I think the main idea is correct, but the examples are not.

Basho

From a quick Google search I found that Farecast had raised $8.5 million in venture capital funding through 2005. It is likely that further amounts were invested. So the returns are not that out of line with any other kind of startup.

Even a successful restaurant (valued at say 5x earnings) could hope for a 10x return over initial capital investment. The work of a few smart people make it possible.

Damon

Given the scalability of Farecast I think Microsoft may have gotten Farecast at a large discount.

http://www.broadcastthoughts.net/2009/08/expected-value-of-information.html

keith

Maybe Ananova.com could have made more money if they went with ANOVA?

miepmiep

Nurses are worth much more to society. The problem is just that no one wants to be a statistician so the saleries are high to get someone do it.

Dennis Rice

I can't seem to figure out how to make it display prices in Canadian dollars.

Andrew

I tend to argee with Miguel @ 1.

I'm a BusinessObjects developer, but I would hardly consider myself a statistician or a number cruncher, per se.; instead I tend to view myself more as an IT professional. These business intelligence tools are used to aggregate, analyze or present data that are not necessarily easy to evaluate directly from the database. You essentially get reports on the specific information you already have.

Farecast, however, is a more traditional statistical model that is used to determine the probability of a certain event occuring (ie. a fare increasing) based on a deep review of prior results.

Guess I just differ on your definitions. But I would bet you that many other in the business intelligence field agree with me.

Scott

Andrew,

That is exactly why the business intelligence field, in general, is not intelligent. It is simply reporting tools. The missing part of most BI is the intelligent analysis that should be occurring one step after what you are describing as your job. (I work in BI as well and think BI really is just IT at this point, when the promise is for it to be so much more).

ben jardine

Nurses are paid less than statisticians simply because a higher proportion of the population are capable of nursing! Simple really, although the "life's unfair" brigade will never accept basic supply and demand economics.

Jean

The business application of statistics typically uses only the most basic statistical techniques and concepts, rarely going beyond Statistics 101. Should businesses dig deeper into the statistical toolkit? You seem ambivalent on this point. On the one hand, you are surprised by the high price paid for Farecast given that "it is just running something close to a very simple multivariate regression." But on the other hand, you conclude, "It goes to show that thinking up the right regression to run can be worth millions."

My hunch is that the simplest techniques are indeed very valuable. First of all, they are a great starting point from having zero statistics. If there are decreasing marginal return of additional analysis, you get the biggest bang for your buck just by taking the first step. Furthermore, it is possible that additional analysis could actually perform worse! Studies show that simpler model often perform better than more complex models under realistic conditions where statistical assumptions don't perfectly hold, data is incomplete, conclusions are generalized to other cases, and conditions change over time. (See, for example, the work of Gerd Gigerenzer on simple heuristics.)

Read more...

pk

I have a question somewhat/not related to the discussion. My friend gets a haircut at a franchise place for roughly $10 once every three months. I get my hair cut from the same lovely lady every two months, without waiting, fear of bad outcome, and generally speaking an excellent experience for approximately $30.

I'm trying to figure out a formula for why it makes sense that spend 3x the amount he does on a haircut.

Please help..

tom kneeland

Did any of these "hot shot", overpaid, statisticians forecast, far cast, or predict the stock market crash of last year, or the oil prices per gallon hitting well above $4.00 12 months short months ago ANSWER: NO! Q.E.D.
None that anyone ever heard from, anyway. So much for this cohort of "geniuses". (see scammers)

Where were they when we needed them 18 months ago? No where to be found!!! IMO.
Now we are to believe that they can predict the future by crunching numbers! Lord please spare us from this silly, scamming, self-serving propaganda. This is expensive baloney!

There is not one of them on the face of the earth who can even tell us with 100% certainty that the Sun will in fact come up tomorrow, period!

Their correct prediction "prior track record" over the last 200 years is beyond pitiful. It is embarrassingly abysmal.
They should be ashamed of their scam.

pd

Tom Kneeland-
I can tell you with 100% certainty that the Sun will come up tomorrow.

And if it doesn't, well, you've got bigger problems on your plate than worrying about statisticians.

Peter Köves

@Tom Kneeland:

you're asking where the statisticians were when the stock market crashed. The answer is: Not in the position to provide the crashes. Statisticians were not high-rolling the savings which they were supposed to take care of.

Your assumption would mean that just bacause weather forecasters are not able to warn against tornados and that New Orleans has been flooded we should get rid of weather prediction alltogether.

I think that too little statistics are being used. I've got a master degree in statistics, having written my thesis about the volatility of stocks. When I apply for a job in a bank where the work mainly consists of using statistical methods the reaction I usually get is that I don't have any "bank experience". This take which seems to stem from Zeno's paradox that Achill will never catch up with the turtle provides some cultural inbreeding that prevents that innovations are not implemented as they could be. (Note: That the guys from the German bank KfW who transfered 319 million Euro to the just broke Lehman Brothers did have their positions then means without doubt that they did have this ominous "bank experience".)

I'm convinced that the world needs more statisticians and more implementation of statistical methods because that's what is needed to save the world and because I'm still looking for a job ...

... you may think but that's wrong. I believe in statistics. And when working as a statistician means being overpaid - well, somebody has to do the job.

Read more...

Abhishek Verma

Implementation means different things in different contexts and to different people. What do we mean by implementation? H.G. Wells once said 'Statistical thinking
will one day be as necessary for efficient citizenship as the ability to read and write'. Recently Harry & Schroeder (2000) remarked 'We believe that statistical knowledge is to the information and technical age what fossil fuel was to the industrial age. In fact, the future of industry depends on an understanding of Statistics'. These statements imply that Statistical Thinking and Methods should become part of the knowledge base of an organization and part of doing business. This is the kind of implementation we are discussing. In other words a business or industrial organization institutes Statistical Thinking in all its functions and the use of statistical tools and data based decisions becomes a part of the every day business.

In this context we are not thinking just about a statistician consulting with a scientist or an engineer for a project. Although such activities are very important, our goal is much broader. Our vision is that statistical thinking and tools should be entrenched in the organization so that they play a prominent role in it's daily activities. In such organizations the roles of statisticians are quite different from the traditional one of helping a client with some data analysis.

Read more...