How Can We Measure Innovation? A Freakonomics Quorum

There’s one theme that we’ve touched on repeatedly in our Times columns and on this blog, and which we’ll devote considerable space to in SuperFreakonomics: how technological innovation and robust markets tend to fix a lot of problems that seem unsolvable.

In the business community, “innovation” is a buzzword of the highest order (so high, in fact, that some people run screaming the moment they hear it mentioned). See, for instance, Fortune‘s innovation blog.

For all the talk, however, there remains the murky issue of how to properly measure innovation. The Department of Commerce, which has begun addressing this challenge, has this to say:

The United States today is more than 75 percent wealthier in terms of real G.D.P. per capita than it was 30 years ago, which is largely attributable to productivity gains driven, in large part, by innovation.

We gathered up a group of people who think about this issue — Ashish Arora, John Seely Brown, Seth Godin, Bill Hildebolt, Daphne Kwon, and Mark Turrell — and asked them directly:

How can a company measure innovation?

Here are their answers. Many thanks for their participation and insights.

Seth Godin, the author of 10 books including Purple Cow and his latest, Meatball Sundae, founder of Squidoo.com, and author of his own blog.

“If we embrace the process, not the event, we win.”

If the question is, “How can a company quantify and measure innovation?” — then it’s certain the company is asking the wrong question.

Innovation happens long before the benefit is realized. And, by definition, each innovation is different than the one before it. As a result, you can watch an innovation become profitable and then say, “Wow that was great, let’s buy some more of that.”

Great organizations have faith in their future, and part of that belief is that innovation pays, even when it doesn’t. It pays because the next innovation — the one after this one — will pay.

And if we embrace the process, not the event, we win.

John Seely Brown, co-chairman of the Deloitte Center for Edge Innovation, former chief scientist of Xerox Corporation and director of its Palo Alto Research Center (P.A.R.C.).

“Metrics for such innovations are tricky, since social practices evolve slowly and surround us continuously, and thus become our frame of reference. It is not until we are abruptly forced to put them aside that we become aware of how much they have been shaping our lives.”

Ask how to measure innovation and you will get a barrage of answers. To help get some purchase on this question, let me suggest we consider three different kinds of innovation, each with its own metrics.

Incremental innovation might be best captured by the saying, “cheaper, thinner, faster and, of course, more features.”

For example, just look at the parade of L.C.D. televisions over the last few years, or digital cameras or cell phones. Incremental innovation is the mainstay of consumer electronics and many other consumer goods. It can be easily measured by looking at how much revenue each year is produced by new products versus legacy products.

These kinds of innovations are both predictable and are important for our economy. Although it is easy to measure the consequence of an incremental innovation, the progenitors behind them are often hard to determine — they can be suppliers, customers, or the company’s own internal R&D.

Architectural innovations are often deeper and more surprising than the incremental kind since they involve a restructuring of the very building blocks of a product family, industry, or infrastructure.

Consider, for example, voice over I.P. (V.O.I.P.) such as is being used by Skype, or the most recent announcement from I.B.M. on their new memory system called racetrack, or Google’s cloud computing. Each of these is changing the game and can be measured by the degree they transform the core competencies required to make, service, and deliver them and the degree they disrupt current business models. Perhaps V.O.I.P. is the clearest example of such.

Disruptive innovations are perhaps the most interesting, at least from a societal point of view. Many students of innovation question if something can even be called a real innovation if it doesn’t end up altering social practices.

Consider, for example, how the cell phone, digital camera or the personal computer, as products — or the world wide web, as a service — have slowly but surely transformed how we live, work, learn, and socialize. These innovations cause us to see and interact with the world differently.

Metrics for such innovations are tricky, since social practices evolve slowly and surround us continuously, and thus become our frame of reference. It is not until we are abruptly forced to put them aside that we become aware of how much they have been shaping our lives.

But the types of innovation we have mentioned thus far miss perhaps the most fundamental kind and the least appreciated kind of innovation — perhaps precisely because these innovations are so fundamental.

Let’s call these institutional innovations, because they actually enable society to function.

In periods of stability they are barely noticed since they are simply considered the way things are. Ask yourself this question:

What innovation over the last several hundred years has led to the most wealth creation?

I was once asked this by a group that was hoping I would say it was the microprocessor. But no, my guess was that it was the creation of the modern corporation (Ltd. in the U.K., Inc. in the U.S.).

The great innovation in the modern corporation was ownership without liability. This allowed shares to be sold on an open market and, unlike other forms of ownership, limited a shareholder’s liability to the price paid for the shares and nothing more. This made it possible for ordinary people to become investors.

My goal here is not to argue that I was right, but rather to suggest that institutional innovations tend to have pervasive and subtle influences on our lives. And in periods of great change, like now, they may well have more impact on us than any other kind.

For example, consider the impact that open source software license B.S.D. used for Linux is having, or the copyleft (institution) used by Wikipedia, or the creative commons licensing regimes, or the global process networks of Li & Fung’s apparel operations around the globe.

These institutional innovations are just the tip of the iceberg of what we will see and need in our age of accelerating change.

Ashish Arora, professor of economics and public policy at the Heinz School, Carnegie Mellon University.

“I can only quote Joseph Schumpeter … and say that ‘the wish was not father to the thought.'”

The creation and application of knowledge is at the core of innovation.

This knowledge need not always be new; Nathan Rosenberg and David Mowery recount how Henry Clay Frick used his knowledge of basic organic chemistry to understand which types of coal were best suited to make steel. Other steel producers at the time relied upon tradition and intuition to select their raw materials, providing Andrew Carnegie (Frick’s partner) with a decisive advantage.

In the twentieth century, American firms in a wide range of industries invested in reducing costs, improving the quality of their products, and in developing entirely new and useful products. As they did so, they also tried to measure what they were getting for their investments.

At one level, therefore, measuring innovation is straightforward.

Innovation can be measured by the additional profit it generates and expressed as a return on the investments made for that purpose. At the level of the industry or even economy as a whole, economists use the concept of “total factor productivity,” which captures a similar idea.

But this is a cop out.

The profits from an innovation can be a long time coming and depend upon many factors — some outside the innovator’s control. An innovation may be ahead of its time (as Charles Babbage‘s mechanical computer perhaps was), may require extensive development, or may require extensive investments in infrastructure (as any clean fuel, such as hydrogen, certainly will).

So, if the reason for measuring innovation is so that a firm can reward its employees, attract investors, and persuade potential customers, we need to measure innovation — rather than the impact of innovation (as captured by return on investment or total factor productivity growth).

Therein lies the problem, for an innovation may be little more than a new business model to satisfy an existing customer need in a new way (McDonald’s, Starbucks, Barnes and Nobel, PepBoys, etc.).

The measurement problem is obvious. But in the spirit of, “It is better to light a candle than to curse the darkness,” firms use a variety of measures.

Some measure the number of new products and services they have introduced, or the share of their revenues linked to these new products and services. This is the measure that the government statistical agencies in Europe use as well in their Community Innovation Surveys (C.I.S.).

This seems sensible, but problems remain. In some cases, the product may be new only to the firm; another firm may have already introduced it. This may be an innovation for the firm, but not for the market. In other cases, the product may be have been developed by another firm, perhaps in another country.

But even more to the point, who is to say what is new, and what is merely old wine in a new bottle, or the same car with a new tail-fin?

Defining innovation using this measure comes down to each firm applying a version of Justice Potter‘s famous test for pornography — “I know it when I see it.” As long as a firm applies the same standard, the number of new goods introduced may be a good way to track whether the firm has become more or less innovative over time.

However, it is not a good way to compare across firms or countries. For instance, the C.I.S. surveys imply that Spain and Portugal are more innovative than Germany, France, and the U.K.

For innovations of a technical nature, industry or government standards can provide another measure:

Indian software outsourcing firms worked to achieve CMM-level 5 certification as a way of signaling the high quality of their software development skills — these standards do not always mean what one might imagine.

Oracle Corporation widely advertised that its database products satisfied the highest level of security standards (popularly known as the Common Criteria) only for hackers to announce, days afterwards, that they had found ways to hack into the product.

Patents are another widely used measure. Typically, patents were thought to apply mainly to technical inventions, and in most of the world, they still do. However, in the United States, one can even patent innovations relating to new ways of doing businesses, as Amazon’s infamous “one-click” payment patent, or State Street’s patent on how to assemble a financial portfolio showed.

In other words, patents may represent land grabs in the intellectual space, rather than innovation, and many researchers are convinced that the patent system has gone too far. It is also true that many — perhaps even most — patents turn out to be of little value, so that the distribution of patent values is highly skewed.

That said, recent research suggests that patents are valuable. Research by Dietmar Harhoff and Alfonso Gambardella and their colleagues, based on a survey of inventors of patents, reveals that the typical (modal) patent was worth $9,000. However, a full 50 percent of the patents were worth at least $450,000!

Other research reveals a somewhat lower range of values, but on the whole, patents are a useful, though imperfect, measure of innovation.

However, for a firm, the problem with using patents to measure innovation remains: is a particular patent a blockbuster or a bust?

The way forward may lie in changing the viewpoint — treating the portfolio of patents as finance firms treat portfolios of assets of uncertain value. Some of the assets will pay off while others will not, and it is difficult, if not impossible to be more certain.

However, one can be more certain about the value of the portfolio. Similarly, firms may be able to borrow techniques from the world of finance.

A promising possibility may lie in the not-too-far-off future:

Innovations and new knowledge are increasingly being traded on the market, via licensing deals. My own research, with Alfonso Gambardella and Andrea Fosfuri, indicates that the value of the deals in the market for technology ranged between $30 billion and $45 billion in the mid 1990’s. More authoritative estimates, produced by Carol Robins of the Department of Commerce, imply that the value of such deals was about $60 billion in 2002.

The idea is to use these payments to estimate the value of a patent portfolio, much as one might use rental payments to estimate the value of a portfolio of real property.

In the end, however, all measures fall well short. This was also the conclusion a recent advisory government committee charged with recommending measures of innovation reached.

Innovation is simply too complex for a single summary measure. If this is too gloomy a conclusion, I can only quote Joseph Schumpeter (to whom we owe the immortal phrase, “creative destruction”), and say that “the wish was not father to the thought.”

ExpoTV.com

Daphne Kwon and Bill Hildebolt, C.E.O. and president, respectively, of ExpoTV.com, which was named the “Most Innovative Company” in the 2007 Stevie Awards for Women in Business.

“Embracing failure … brings you dangerously close to failure’s more deadly cousin, flailing.”

“It’s such a fine line between stupid and clever,” said David St. Hubbins (of Spinal Tap fame).

As a start-up company trying to pioneer a new industry, measuring innovation poses an especially tough challenge. Success by traditional metrics — revenue and profits — only becomes evident after you’ve become successful. In fact, early innovation often looks like idiocy and idiocy can masquerade as innovation.

Why is this?

According to Clayton Christensen in The Innovator’s Solution, truly disruptive technologies are generally worse than existing incumbent products at launch.

When the first digital cameras were made available to the public, their picture quality was far inferior to film cameras. Today it seems obvious that digital cameras were destined to vanquish film, but in the moment, it was hard to measure the true innovation that was occurring.

Our company’s focus, video product reviews, fares poorly against the incumbent — the ubiquitous written product review — in quality and quantity. But we have a vision that marrying the power of video storytelling with authentic consumer experiences can unleash word-of-mouth influence beyond anything available today.

While we track traditional industry metrics such as number of reviews, breadth of catalog, and quality of information, we’ve added new metrics that help define the goals of consumer word-of-mouth. Defining these new benchmarks helps us select new risk-taking projects that can speed us along our path to success.

How can our experience measuring innovation in the moment (rather than just looking backwards) be generalized for other entrepreneurs and managers?

We’re going to go on record and say that it is all about looking for and then celebrating the unique “failure metrics” in your business:

1) The rate of failure. As noted, innovation’s first steps are likely to be tentative and wrong. In a society that shuns failure, management needs to make a conscious effort to identify and embrace risk-taking. Failures can be turned into learning opportunities, but need to be meticulously constructed to do so.

Postmortem meetings reviewing projects that failed to meet their goals must be designed to be free of finger-pointing and dis-ownership. Instead, management should very purposefully uncover what was learned about its customers or partners through the effort. New projects should be born from those learnings immediately. Iteration is critical.

Most importantly, employees who were daring enough to experiment should be recognized and further empowered to pursue their initial hypothesis in a more informed way.

2) Failing along the right path. Embracing failure, however, brings you dangerously close to failure’s more deadly cousin, flailing.

Every entrepreneur will tell you that their first business plan was not where the company ended up. Every definition of innovation focuses on the fact that change is the critical element. However, empowering any idea for change can clog up precious resources and prevent truly innovative projects from seeing the light of day.

The critical ability of management to separate the wheat from the chaff is based largely on the strength of the company’s vision. Ensuring that management is in vision lock on the larger company mission will allow executives to have a positive, healthy debate about which ideas are likely to make the greatest impact on the company’s success.

3) The source of failures. Another measure we use to determine if our company is embracing failures is whether new strategic ideas are coming from all levels of the company.

Are only executives contributing to new projects, or are they bubbling up from the tech developers, customer service staf, and sales team? First, broad participation can indicate that you’ve been successful in creating a culture of risk-taking. Second, if most of the new ideas are strategic in nature, it can mean that you are effectively diffusing the broad vision of the company to all levels.

We review our board presentations with our entire staff to ensure that they are as aware of the discussions occurring behind closed doors as they are of those with their own managers. Knowledge is power in any organization and diffusing knowledge is essential to releasing creativity at all levels.

Failure must, of course, give way to success in order for customers, not to mention your staff and investors, to benefit. How entrepreneurs maintain the balance between pushing for success and embracing “failure” will be a key determinant in how quickly their start-ups can innovate new industries.

Mark Turrell, the C.E.O. of Imaginatik, and author of the blog Innovation BBL.

“Some of the largest R&D spenders live in a happy world where the number of patents goes up, whilst the returns slope downwards.”

The question of how to measure innovation is linked to, “Why measure innovation?” This differs depending on your viewpoint.

As an investor, you might wish to know that the company is innovating to justify a higher share price — set by market expectations of future growth — itself driven in part by innovation. Deloitte has published reports on the Innovation Premium that measures the contribution innovation makes to a company’s share price.

As an executive, you may need to use an innovation measure as a surrogate of future growth, both in terms of potential and pipeline deliverables.

As a functional director, you may be tasked with measuring innovation at group level, to support overall company objectives, and to assess the growth potential of your own areas.

And finally, back to the macro level and beyond, governments wish to measure innovation to determine the right levels of investment in education, R&D, and sponsoring champions to grow their economies and the economic well-being of their people.

Now, rather than giving the cop out answer of “it depends,” I’d like to try a different, more controversial tack — just to wake people up a bit:

Let’s start with a few commonly used measures of innovation — the number of patents and percentage of revenue from new products — and do a deep dive into them to see if they live up to their expectations.

Number of Patents

The theory of tracking number of patents, I hazard a guess, is that they are one of the few tangible measures of output from an R&D process. The granting of a patent is a milestone in the public domain, and has the added advantage that the granting takes place via a third party, not an executive committee that may be open to internal bias.

The theory would then have it that more patents are better than fewer new patents.

Unfortunately the theory is largely hokum, a fact acknowledged at least tacitly in the bars adjoining R&D conference venues, if not in public.

I recall a Motorola researcher commenting in a recent article that he was effectively coerced into filing multiple patents on trivial components, such as battery catches — an act that yielded zero business value but served to boost the patent numbers. Several leading companies have trawled through their patent portfolios to work out which patents actually drive business value, and are frequently shocked to discover that less than 10 percent of the patents drive 90 percent of the value.

R&D resources, already woeful in their returns (according to studies, 70 percent plus in R&D dollars are wasted), are directed to delivering patents instead of developing high-impact solutions that add value to customers and deliver positive financial returns. And some of the largest R&D spenders live in a happy world where the number of patents goes up, whilst the returns slope downwards.

Percentage of Revenue from Products Less Than 3 to 5 Years Old

This is one of the most commonly used measures of innovation for public companies. It is also highly dangerous, creating systematic distortions in behavior that can destroy business value and any semblance of an innovation culture.

It is supposed to work by calculating the revenue generated from recently introduced product lines, and measuring that revenue as a percentage of overall revenue. It is intended as a form of freshness indicator for the business, and has the benefit in that it measures realized outputs rather than expected benefit.

As a practitioner with ten years experience, and a personal leitmotif of “I break things”, I like to look at measures like this with a view to breaking them.

As a thought experiment, I set out four years ago to work out how a firm or group of people could cheat the metric. I then had the good fortune to meet folks working in the best companies in the world, and ask them whether they had witnessed dysfunctional behavior in their own careers (obviously, not them themselves, of course not, no, not never).

What emerged was shocking. One globally renowned corporate inventor referenced to his company’s practice of S.K.U. Innovation. (An S.K.U. is a shop keeping unit, a code used to track products.) When the company was faced with a shortfall in genuine new revenue from new products, certain groups just added new S.K.U. codes to the accounting system, thereby creating an artificial stream of “new” products.

Another executive in a sporting goods company commented that new products rarely reach anything like peak revenue in three years from initial launch, and so the top marketing executives would time their entry and exit into products based on the target and time line (sound familiar?).

Yet another executive in a manufactured goods company complained that the target could be hit by introducing new products that cannibalized existing revenue streams with worse margins — destroying the profitability of the business.

There is hope, however, of a more enlightened approach to this specific metric. Statements in annual reports have a Sarbanes-Oxley dimension to them, which means that auditors need to be comfortable that these are true statements. Whirlpool, one of the pioneers in this area, took this as an opportunity a few years ago to apply significantly more rigor to their measurement and reporting of new revenue. To save you wondering what they did for too long, the basic model they adopted was as follows:

– Introduce a strict definition for “innovative revenue” that precludes things like S.K.U. innovation.

– Allow new revenue streams to be counted as “innovation” for as long as the stream is growing at more than 10 percent per year (i.e. a growing product could be 20 years old, and providing that it is still growing, the group would still get credit).

– Force new products to have margins at least as good as existing margins in the category.

The two examples above highlight some of the challenges faced by organizations in measuring innovation. These common metrics both have significant negative implications to the efficient running of an innovative business.

There are frameworks that allow firms to better measure innovation. My firm, Imaginatik, has worked on innovation metrics for several firms. Our model is based around three areas of innovation metric:

1. Realized Benefits – a measure of outputs from the innovation process, using the Whirlpool definitions as a reference point.

2. Expected Benefits – a measure of pipeline value of project candidates.

3. Process & Engagement Metrics – a set of measures that describe how the front-end of the innovation process is working, including the level of employee engagement, the number of quality ideas generated, and the scope of the innovation initiative across a wide range of business activities.

Ultimately companies need to craft their own metrics, based on simple models as described above.

One final thought on metrics:

Metrics lose their value quickly if there are no targets associated with them. Measuring for measuring sake is a good way to tie up consultants for a few months, but it does not drive behavior and results. Targets are essential to help drive behavior, and target setting is a critical task for senior management. The absence of targets leads to uncertainty and an inconsistent application of any improvement activities.