When Does Experimentation Matter?

Boy, did I have an interesting conversation with Clay Christensen last week. Our BIF Research Advisor’s seminal work on disruption, The Innovator’s Dilemma, turned 10 this year. Interestingly, I found out that it’s only in the last year that he feels is ideas have finally caught on to become broadly understood and applied beyond the technology industry.

Christensen has consulted to a number of industry giants including, Kodak, Intel and Intuit. After 10 years, he has accumulated many successful case studies. The firm he co-founded, Innosight—which is also a BIF member—continues his work.

We talked about a lot of different things, including his belief that every academic is in some way a consultant. And part of the consulting equation is a training engagement. This, he says, is where he sees a flaw with current innovation practices. “You don’t change a company by giving them ideas. You change them by training them to think a different way,” he said to me. (More on this topic in a later blog entry.)

What’s been sticking in my head though is a conversation we had about the role experimentation plays in the innovation process.

When the Innovator’s Solution was first published, Scott Cook, chairman of Intuit, gave a copy of the book to each of his 50 senior managers and made them read it. He then divided the group into teams of 3 and told them to read the book again. Each group was then asked to analyze the history of a particular innovation project. The assignment was to explain why the project either succeeded or failed through the lens of Clay’s disruptive model. Or, were there other factors that drove the success/failure that Clay hadn’t touched upon yet?

For Clay, he expected the real value would come from discovering what the teams couldn’t explain through his models. Turned out that the teams could explain 100% of the success/failure variants by virtue of the model. He told me he was really stunned. “Disappointing in a way, because the only way to improve theory is to figure out what your theory doesn’t explain.”

I found his next example more striking.

Intel has a group called the New Business Initiatives Group. Basically, if someone has an idea for a new growth product, they submit a proposal to this group. Intel hired a Christensen disciple (my term, not his) who had schooled himself on all of Clay’s theories from the Dilemma to the Solution to What’s Next. Clay said he knew his research better than he did. So this guy joins the new business group and before he knew what was happening around him, he learned that Intel had funded 41 new business initiatives (I don’t knowwithin what timeframe). He decided he was going to run an experiment, so he took all 41 proposals that had been submitted for funding, read them and, based on Clay’s research, divided them into the ones he predicted would be successful or not. In the end, 7 of the 41 had become successful – which is roughly the success rate of VCs. The kicker though is that the new guy had predicted 6 of the 7 successes and 33 of the 35 unsuccessful initiatives simply by analyzing the proposals through Clay’s lens. WOW!

Theory is a statement of cause and effect. Clay said that without good theory, experimentation really matters because you have to try a lot of different things hoping that they’ll stick. Conversely, if you have a good theory, you don’t have to experiment as much on which projects or new businesses have potential. The experimentation then relates to learning how to execute the strategy that the theory says is going to be successful.

So getting back to the Intel example, based on the new guy’s initial research, since he predicted the outcomes of 38 out of the 41 projects, it’s really only 3 projects that required some first stage experiments.

Further, Clay said the ones that he predicted would fail were not all intrinsically destined to failure. It’s just that the strategy/business plan it was embedded in didn’t match with the theory - but you can take the same idea and revise the business plan so that it maps into sound theory.

Turns out that good theory is a very useful thing. Clays told me that false principles have been taught for a long time. “Academics are poorly trained in how to build robust theories that are useful,” he told me. He’s currently writing a new book looking at public education through lens of his research. It starts out with a description of what good theory is and what “crummy theory” has evolved from schools of education.

Clay Christensen –who is also a BIF Research Advisor- will be speaking at our upcoming summit on October 10th and 11th. He’s a born teacher. And not to be missed.

RELATED LINKS:

• Learn how you can participate at BIF-3.