The downside of applying lean startup principles

The Lean Startup is a bestselling business book by Eric Ries that has become a global revolution with followers in 17 countries. Its principles have been adopted by early-stage ventures, Fortune 500 companies and even the U.S. government. Its core principle is to test a “minimum viable product” early and gather information to further fine-tune development. The idea is to mitigate risk in the product development process.

But there is a downside to early market testing, according to a research paper titled, “Experimentation, Learning and Appropriability in Early-Stage Ventures,” by Andrea Contigiani, a researcher at the Mack Institute for Innovation Management and a Wharton doctoral candidate. By releasing early versions of products, companies risk imitation by a competitor. He spoke to Knowledge@Wharton about when early market testing can hurt more than help.

An edited transcript of the conversation follows.

Knowledge@Wharton: Why did you choose this particular topic?

Andrea Contigiani: I have always been interested in questions around innovation, so when I was choosing a topic for my dissertation, I wanted to find something that would be really impactful, that practitioners and industry people would care about.

I went out and tried to interview as many people as I could, both in the big Wharton alumni community, and also in the Philadelphia startup scene.… It was really clear that most people cared a lot about being a lean startup. Things like experimenting, creating a minimum viable product, pivoting. These were some of the things that came up most.

I was very familiar with The Lean Startup idea, but I don’t think I was fully aware of the impact that it had on business and entrepreneurship. It has really been a little bit of a revolution. The U.S. government is even asking the companies they fund to use lean startup methodology through a program called I-Corps.

The key words in Lean have really become part of the business language in all sectors, not just startups. This has really been an important change, and so I wanted to look into this topic [and validate it] with data.

Knowledge@Wharton: Just to back up a little bit, you mentioned the book, The Lean Startup. Can you explain briefly what that is for our listeners?

Contigiani: Absolutely. The Lean Startup [describes] a way to do business. It applies not only to early-stage companies, but also to everybody doing innovation including large firms, nonprofits and even governments.

The core of The Lean Startup is essentially [to do early] experimentation rather than trying to understand the details of your business, your product, before executing. The idea is to … get market feedback as soon as possible…. There are some specific practical tools that The Lean Startup suggests, primarily minimum viable product (an early stage of your product through which you can get feedback) and pivoting (changing the product after getting user feedback). These are its two big innovations. In general, this [idea] helps companies innovate more effectively. … Much of The Lean Startup [methodology] comes from the amazing work that [entrepreneurs] Steve Blank and Eric Ries have done over the past 10, 15 years.

I attended a Lean Startup conference last year in San Francisco where Eric presented his latest book, The Startup Way. The big change from his previous book … was that now he tries to teach how to do lean innovation to all kinds of firms, not just startups. [Lean principles have led to], I think, a fundamental change in business.

Knowledge@Wharton: I understand that your research actually took those principles and went a step further. So walk us through what the goal of your research was, and what were some of the parameters of your study.

Contigiani: That was exactly what I am hoping to do with my research. [Lean comprises] an extraordinarily valuable set of ideas. I tried to take those ideas to see what the data said about them. While I really believe that Lean has some fundamental value, there are probably some of what we would call in research “boundary conditions.”

There could be situations where going lean is optimal, and other situations where it’s not, and maybe in those situations if you want to go lean you have to also take some other strategies to make that work. And so I wanted to try to understand those boundary conditions.

The first step I took … was to look at a very important concept that in the innovation academic literature has been discussed for a long time: appropriability. [With that view,] I explored, for empirical purposes, a very interesting change in the intellectual property environment in the U.S., which was [the impact of] a U.S. Supreme Court decision in 2014 — Alice Corp v. CLS Bank International. (Alice made patenting in parts of the software industry, mostly business software, less effective.)

This was an important [decision] because it changed the way we thought about patents in a segment of the software industry. It affected some companies doing software, but not all companies doing software. What it did was really changed appropriability for some of the companies. Studying what companies did in that context helped us understand how appropriability plays out in experimentation.

Knowledge@Wharton: Can you explain what you mean by appropriability, and also the terms formal intellectual property, and informal intellectual property, which you mention in your paper?

Contigiani: Appropriability is one of the most important concepts that we have in the innovation academic literature. It comes out of the work of David Teece, a very well-known strategy scholar from U.C. Berkeley. A lot of work that came before and also after contributed to that [body of thought].

Appropriability is essentially the capacity a firm has to appropriate [or retain the] value from its innovation. If you are an innovator — whether you’re a large firm, small firm or inventor — when you create an innovation, the value of that could go to you or to other players in the market, depending on how the market is structured and on the strategy that you take and how the companies react. Appropriability is essentially how much of the value you can keep for yourself, how much you can appropriate.

I thought this could be an important boundary condition for experimentation for going lean. The basic hypothesis … is that when appropriability is not high [i.e., you are not clearly able to appropriate most of the value of your innovation because you have some kind of defenses in place], when you experiment you are giving out some information about your product at a time when you are still vulnerable.

You could get imitated; you could get replicated. Those processes reduce appropriability. So experimentation, if you don’t have an appropriate defensive strategy, might reduce your appropriability. Of course, a fundamental role is played by what we call intellectual property (IP). Some people would even say that IP is what defines appropriability.

There are two important forms of intellectual property. One is what we call formal intellectual property — legal tools to protect your intellectual properties such as patents, trademarks, copyrights and to some extent trade secrecy. While some would say trade secrecy is not necessarily a formal tool … it is regulated by law. Informal intellectual property is a set of strategic tools, strategic moves you can take to protect your innovation.… Things like a complex product, big lead times [and other] bars to imitation.

This is one of the aspects of the study.

Knowledge@Wharton: What are the main takeaways from your research? How can startups take what you have found and apply it practically to their business?

Contigiani: I basically hand collected data on the product development processes of 1,200 software startups [using] online sources such as tech magazines and social media. I [tracked the firms] to see when they were testing or experimenting and when they were essentially launching their products — actually entering the market. I wanted to see how the overall strategy of these companies changed when this court decision, Alice, took place because it essentially made patenting [formal intellectual property] much harder for some companies.

I found two things, broadly speaking. One is that after Alice took place, the affected companies changed strategies. In particular, they seem to be less likely to test their product; they do less experimentation. Given that they can no longer use patents to protect their ideas, experimentation becomes risky. Sure, you get the benefit of learning, but the cost of doing so goes up. So it’s not necessarily a good idea.

On the other hand, they seem to launch their product faster. They go to market sooner and so they can start getting feedback. Adapting your product, or pivoting, once you are in the market is a little harder [than early testing], because adaptation costs are higher. But on the other hand, once you do that you are essentially creating other barriers to imitation, like brand and network effects. So it is safer.

The second result is more about performance, and there I looked at what happened to companies that did a lot of experimentation while in the new post-Alice regime where they could not really protect through patents. I saw a negative correlation between doing that and performance.

Companies that experimented a lot without potential access to patent protection were less likely to get funding, and they also seemed less likely to get acquired. And so overall, this choice seemed to really affect the performance.

The [practical] implication for startups is that in very uncertain markets … when you choose that strategy [of market experimentation] you really have to take into account how you can protect your invention…. It is really important to keep an eye on intellectual property and appropriability because it can really hurt you if you don’t. By talking to entrepreneurs, my sense is that they are really focused on learning, less so on appropriability.

There are also implications for other players in the economy, so I will briefly talk about those. I think this matters to investors. Investors seemed to be really aware of this trade-off, because I saw companies that didn’t seem to worry about appropriability get less funding. [But then perhaps they’re investing] in companies that needed a lot of learning because they are building something very new, very novel. If so, you might need a lot of experimentation, but [you also] will need to protect yourself through IP effectively.

I want to end with implications for public policy, because that may matter as well. Policy makers obviously care a lot about innovation. Now if in important markets or sub-sectors of the economy it is really hard to protect through IP, [companies in sectors where experimentation is important] might be less likely to succeed. On the other hand, companies might realize that actually it is hard to try to build innovative products without being able to safely experiment, so they might [stop experimenting] and try to actually compete in more traditional markets. That would mean less innovation for the economy.

Knowledge@Wharton: What sets your research apart from prior work in this area?

Contigiani: What I was trying to do was to really understand this idea of experimentation, and more broadly understand the lean methodology, which I think is a fundamental innovation of this past decade. There are a few other researchers around the world doing research on this. But I’d like to think that at least I tried to differentiate my work in a couple of ways.

I put together a detailed data set of how companies test, when companies launch, how companies do afterwards…. Most other studies on this use very small data sets, or mostly they look at case studies. Those approaches have very important advantages as well, but obviously it is hard to do statistical inference without large samples.

Putting together this large sample was a big effort. I really want to mention that I worked with an amazing team of research assistants, and they were partly Wharton undergrads, and partly people around the world that I hired.

I would also mention a couple of other things that are more academic, and in some sense, I think have some value for research. One is that I haven’t seen much other work trying to examine or trying to use the Alice Supreme Court decision. That was such an important event, and I think research should really look at it. In my area, I think this is the first study that does that.

Also, there are two important streams of research in management strategy: research on organization learning, research on appropriability. [Current literature] looks at two sides of this trade-off on experimentation, [but they] haven’t been combined too much. My work does try to connect them.

Knowledge@Wharton: How will you follow up this research? What’s next?

Contigiani: I definitely want to keep going and understand a little more of what are the boundary conditions of experimentation — in startups primarily, but maybe also in large firms and to some extent in governments and [public] policy. At all of those levels people innovate, and maybe we can try to see how and when experimentation makes innovation better and more effective.

There are a couple of things that I really want to work on soon. There are at least two other boundary conditions of experimentation that I think we need to think about. One is reputation. When you go to market very early with a minimum viable product — with an early stage product, with a prototype — that is super helpful because it helps you get market feedback and learn.

On the other hand, if the feedback is negative, while it’s useful because you can pivot early on, that feedback might affect your reputation.… It doesn’t seem to me that entrepreneurs and maybe even investors consider [risks to reputation] carefully. I want look at the data and see how it plays out.

There was a really nice study from Duke University and from Harvard University a few years ago that looked at how rating systems affect the way companies behave and strategize. I think rating systems are a very nice empirical tool to measure reputation. So I want to see how experimentation plays out in some sectors where rating systems are more or less strong, or more or less organized, to try to understand the reputational concerns of experimentation.

The second piece is really looking at the learning process itself. Experimentation lets you learn but may have these other costs like imitation or reputation. But actually learning is to some extent the most fundamental piece here. Now there is a lot of research in strategy that shows that learning is hard to do. You can get information, but you are not necessarily learning the right thing.

That’s because there are many types of biases which arise both at the individual level — people learn in different ways — and at the collective level, at the team level. It depends on the structure of the team, the experience of the team, and how the team works together. We need to look at the learning process and see how we can make that work optimally to make experimentation work well.