Introduction

The Lean Startup (LS) is a popular strategy to help entrepreneurs improve their odds of success during their search to identify opportunities and to create a venture. LS has a considerable following. Startups in the software space and the National Science Foundation’s Innovation Corps™ program utilize this methodology [1,2]. This framework can aid entrepreneurs in handling uncertainty, learning, and making decisions. It can be boiled down to innovation through repeated hypothesis-driven experimentation to define a product (service) and a business model that a firm can scale [1,3-5].

LS involves two distinct phases (Figure 1) [1]. The first focuses on identifying customers and searching for customer needs (also known as customer identification or development) and then realizing product/market fit (P/MF) and a repeatable sales model (customer validation). The second involves company creation, where the focus changes from learning to scaling. LS involves learning from customer interviews, research, and testing of ideas, with a clear and explicit emphasis on experimentation with a Minimum Viable Product (MVP) to gauge traction through lean’s build-measure-learn approach [5]. This effort aids the startup team to efficiently make “go forward” or “fail fast” decisions. It also provides the opportunity for a startup firm to develop dynamic capabilities, which Teece and colleagues define as a firm’s ability to integrate, build, and reconfigure internal and external competences to address rapidly changing environments [6].

The literature does provide some theoretical foundation to support LS practices. Frederickson and Brem identified evidence (based on publication quantity and quality) that supported specific LS practices (Table 1) [7]. These included: (1) user and customer involvement in product and business development- very strong; (2) iterative approach to new product development and effectual thinking (planning versus doing)- strong; and experimentation with product development and the use of a minimum- medium.

Ladd provided further supportive evidence based on his work at a cleantech accelerator [8]. He evaluated teams that used LS methods and tools versus those teams that did not. Judges, who were industry experts, evaluated pitch competition performance, which was the basis for defining a successful outcome [9]. He observed that LS teams, which had focused their testing on the target customer segment, value proposition, and channel outperformed the non-LS teams by a factor of two. Ladd also found that groups that clarified and tested hypotheses about their businesses performed three times better than those who did not evaluate any such question [9].

While some evidence appears to support the methodology, such observations are not conclusive that LS offers an unequivocal framework for startups to be successful. Furthermore, a systematic literature review, The Lean Startup: A Pragmatic View on Its Flaws and Pitfalls, by Heitman indicated that little evidence existed to support (or deny) practical success of the LS methodology or to justify the hype within the entrepreneurial community He did not see it as suitable for every startup as a map to success [10].

Considering the attention around LS, the aim of this paper is to explore the limitations of this methodology based on published literature to identify what issues do exist that entrepreneurs should be aware of when using LS as part of their startup journey.

Methods

This paper reviews relevant literature focused on LS and its limitations. Research efforts focused on multiple electronic databases (ABI/Inform, JSTOR, Google [and Google Scholar], ProQuest Dissertations and Theses, Science Direct, and Web of Science) to identify relevant publications concerning LS and its limitations. Search terms used included “Limitations,” “Issues,” or “Problems,” and “Lean startup.”

Interestingly, initial efforts captured 200 citations. These included published peer-review papers and non-peer review documents (e.g., graduate theses, business publications, blogs, and non-peer reviewed web content). Most of the publications involved those from advocates and practitioners describing the methodology. A closer examination led to the identification of a limited number of publications that specifically looked at LS and its limitations. These works were predominantly in the non-peer review space and included trade publications and blogs describing the methodology.

Issues with Implementation

Multiple individuals highlighted inconsistent experience with the methodology [8,11-14]. Most of these reports detailed their individual experiences utilizing case review, experiment, interview, and survey methods.

Ladd provided the most structured evidence with an experiment he conducted in a cleantech accelerator using performance in a pitch competition as the endpoint [9]. While he did observe some success based on the comparative performance of several of the LS teams, he observed some limitations to the methodology. He found that some of the LS teams in his study did not perform as well. In fact, he saw diminishing returns with some of the teams that had continued to test multiple hypotheses. He noted that teams those who used both open-ended conversations/interviews and formalized experiments did worse in the competition that those that did either one or the other during the early stages of discovery. He also observed that continued pivoting might lead to a disheartened or impatient founder. Finally, he noted that LS does not necessarily define which aspects of the business models the entrepreneur should address first.

Blomberg shared his individual experience with LS in a personal startup [11]. He identified limitations, highlighted what would make the lean process more productive, and outlined areas for future research. He pointed out several essential observations: 1) question one’s assumptions; 2) do not assume one is talking to the right customers; 3) iterate rapidly; 4) iteration was not pivoting; and 5) build virtual versus physical prototypes for MVP testing.

Gustafsson and Qvillberg examined lean methods in an early-phase manufacturer [12]. They tested lean in an early-phase manufacturer with a new technology facing a high degree of uncertainty about customer needs and potential applications. In this experience, they identified critical issues with this process: 1) facing challenges in getting rapid feedback on a physical product; 2) pivoting due to the lack of big problems seen by customers; 3) developing a scalable business model; 4) developing a physical MVP; and 5) accessing the right customers for interviews.

Juréen evaluated LS as part of a healthcare project focused on self-managed post-stroke rehabilitation [13]. She found that in the health care setting, the LS approach was not applicable in the early stage of the project. She felt that this finding was due to the broad range of participants in the health care process, which involves several different stakeholders. She noted that this diversity of stakeholders conflicted with lean’s narrow and agile focus. She also highlighted conditions for success, many of which reinforced the principles of LS: 1) involve different types of users from the start; 2) implement both a need-and business model-driven approach; 3) engage users as a valuable source of information; 4) get out of the building; 5) employ a strategic push; and 6) involve multidisciplinary expertise.

Nilsen and Ramm conducted a quantitative survey-based study of Norwegian high-tech startups [14]. Their findings suggested that the use of the lean framework might not correlate with success. They obtained responses from 47 of 90 companies contacted. About familiarity with LS, 42.7% of the respondents responded by selecting “7” (on the 7-point Likert scale), with a mean of 5.62. Respondents rated their use of LS methods an average of 1.13, with a range -3 to 3.77. What was striking was that in running simple correlations, Nilsen and Ramm did not find any significance between awareness and use (r=0.093, p=0.535) or between the use and success (r=0.091, p=0.542), based on a composite score of company attributes that the survey captured.

The business-to-business case study using LS in Indonesia by Nirwan and Dhewanto reinforced prior observations related to implementation issues with the LS approach [15]. The authors identified several barriers when evaluating the process by creating and validating the problem and then the solution. A notable hurdle involved the ability to access customers for interviews. This challenge made it difficult for the entrepreneur to capture customer feedback and to confirm hypotheses. Another issue involved difficulty in pivoting due to the lack of big problems; the problem in the case study was minor, and the solution was an incremental product, in which customers had interest and competition was fierce. A further barrier involved the speed of iteration due to regulatory and administrative considerations involved with contacting customers. Some of these issues may be idiosyncratic to the environment; however, it points to the challenges concerning the implementation LS consistently on a global basis and in conducting the customer development process to identify customer needs.

Problems with Hypothesis Development and Experimentation

Multiple investigators raised issues concerning experimentation. Shafer identified several potential pitfalls [16]. The first involved facilitator and observer bias concerning hypothesis development and testing. The second involved ambiguous results from open-ended experiments. The third consideration was the lack of statistically significant effects due to small samples.

Interestingly, Frederikson and Brem identified several gaps in supporting part of the framework [7]. These authors highlighted concern related to experimentation, including the emphasis on testing over long-term planning, which may limit the scaling and design of experimentations, an area is for further research [7].

Furthermore, Ladd did not find any linear relationship between the number of validated hypotheses and a team’s subsequent success [9]. In fact, he saw diminishing returns with LS. Ladd added that LS might produce “false negatives,” translating to the rejection of good ideas because the strategy does not provide a clear rule for defining success (P/MF), stop testing, and initiate scaling. He also observed that the LS teams did not have a clear threshold for go/no-go decisions based on their experiments.

Grace Ng, a co-founder of Javelin.com (a software and services company that utilizes LS), highlighted several pitfalls in her 2014 Forbes Entrepreneurs piece [17]. The first involved testing the wrong thing. She noted that the “I have an idea!” hypothesis would lead to tunnel vision and, thus, the entrepreneur would not identify whether this guess was correct. She continued that forming the wrong hypothesis is the result of the entrepreneur misunderstanding the problem and overlooking the root cause. Another issue Ng saw was the inability to define a baseline metric that the firm could use to establish accountability and success.

Challenges Associated with the Minimum Viable Product

Ng observed that proper MVP development was another concern [17]. She noted that several startups she has worked with dismissed the need for building an MVP that would test a firm’s riskiest assumptions and shorten time to get feedback. Instead, Ng saw that many times entrepreneurs already had ideas that were already products or solutions and these were trying to find a problem or market to address.

Heitmann highlighted that problems existed with the MVP concept [10]. He argued that bringing an inferior, unfinished product to the market (e.g., “Buggy” piece of software) leads to a considerable percentage (96%) of dissatisfied customers, who do not provide direct feedback to the startup because the customers may be overwhelmed by the incompleteness of the product concept. Also, Heitmann pointed out that the adding and testing of new features one by one can lead to an unnecessary testing loop that wastes both money and time. He continued by questioning whether LS was worth the rush. Instead of an MVP, he advocated that entrepreneurs focus on the concept of the Minimum Desirable Product (MDP), which he defined as having a product that has to produce enough desire and satisfaction for the customer to remain interested and not abandon the MVP for initial release [10]. Heitmann raised the concern related to rather than concerning looking through the keyhole during early research with only early adopters. Thus, this focus would miss those in the “early majority” segment, which he felt was essential to scaling a business. Heitmann continued that without learning and change, the previous work was for naught. He added that lean followers who may not fully invest themselves into fully understanding and integrating Ries’ and Blank’s teachings do set themselves up for false expectations.

Nirwan and Dhewanto observed that the MVP was confusing the implementation of the process [15]. They added that while the MVP’s purpose was to create a minimum product that would capture customer interest, the startup did not want to create an inferior product (a common concern shared by many entrepreneurs), and yet it could not afford to go too far in developing the full product due to limited resources.

Rao provided an Indian perspective on the critical role of the MVP. The first issue referred to a limited Intellectual Property (IP) base that protects lead-time or early-mover advantage that fast innovations possess that the MVP might not embody [18]. The second involved a low budget for an MVP. Hence, Rao pointed to the fact that startups plan to enter the Indian market only after acceptance abroad and a hybrid model with one leg in Silicon Valley, and another in Bangalore is notable. He noted that they see demanding customers in India used to ‘readymade’ products from the West and were well on their way down the adoption and commercialization curve but have never experience innovative early-stage products from local startups.

Kortman (2012) added a further perspective on the problem with the MVP [19]. He shared the example within his own company, ThingShare (a peer-to-peer video game renting platform), of its struggle with the concept. He explained that while the company had invested time and expense in going to market with more than an MVP, the product was no more than smoke and mirrors. To this end, he felt that the company short-changed its early adopters, resulting in the need to apologize to these individuals for an incomplete product or one that overlooked early user feedback. Kortman wondered whether the company “launched” the product prematurely. In closing his discussion, he raised the question concerning the point at which an MVP would be truly viable: (1) critical mass of customers; (2) no more features needed; (3) revenue; or (4) profitability Ultimately, he defined viable in MVP as the ability for the platform to sustain on its own.

Biases and Implementation Issues with Interviewing

Ng pointed out that entrepreneurs tend to ask the wrong questions. This practice is a result of the fact that the startup has set its discovery phase to be about selling the product, rather than going out and interviewing the customers to discover current behaviors and insights to help the entrepreneur pivot to an appropriate solution. The common mistakes that she observed included leading questions, selling too soon, talking too much, and not digging deeper when insights surface. Finally, Ng’s fifth issue involved problems that occurred during the pivot phase. These included vs including the discarding of an idea without learning from the data and getting the whole team on the same page.

Related to Ng’s concern about asking the right questions is that of interview biases. York and Danes highlighted that entrepreneur biases, along with heuristics, exert significant risk in the customer discovery process [20]. They observed that entrepreneurs, who are overly active and constrained by time and expenses fail to obtain or notice available information critical for making a proper decision. York and Danes continued that many entrepreneurial activities rely upon the subjective view and limited data.

In reviewing the literature, York and Danes identified several biases and heuristics that carry significant risk to the discovery or customer development process [20-24]. These biases include those of: (1) selection (i.e., friends and family); (2) confirmation (i.e., leading, confirmation, closed-ended questions); (3) overconfidence (i.e., overestimation of knowledge, skills, and precision of one’s data); (4) optimism (i.e., unlikeliness to experience failure); (5) representativeness (i.e., generalization of findings from small samples); and (6) acquiescence (i.e., respondents offering answers they think the entrepreneur wants to hear rather than genuine opinions).

In a related paper, Chen and colleagues examined the cognitive biases that can influence the entrepreneurs’ judgment of P/MF [25]. The authors highlighted several of these biases when adopting the getting-out-of-the-building approach, search characteristics. These included: (1) for face-to-face, interviews that of saliency, vividness, and inappropriate cues; or analogy; (2) for sequential interactions, that of recency and primacy, and that of contrast effect; (3) for large samples the effects of overconfidence (redundancy) and dilution; and (4) for the entrepreneur, that of biased processing. The authors noted that the essential point of the interview was to explore, not validate. In recognizing that while biases could be challenging to eliminate, they suggested techniques that could be employed to reduce the effect of such predispositions when interviewing customers.

Not the Correct Model for Breakthroughs

Kressel and Winarsky provided evidence as to why the LS was not right for breakthrough ventures [26]. They pointed out how the LS approach might make sense for software- or web-related companies with modest startup operating expenses. They advocated that tech specialists employ the “pivot until you succeed” approach best as a solution to a defined market problem. They quoted a Sequoia Partners venture capitalist, who indicated that pivoting means failure and likens constant pivoting to having “a compass without a bearing.” They added that the “fail-fast“ approach made sense only to investors who spread their risk over many small ventures, in small bets, and not to the entrepreneur who bears the expense (personal, professional, and financial) from failure.

They felt that the LS approach did not make sense and fell short of what “world changing” products or critical components for success associated with billion-dollar breakthrough ventures require [26]. These include those products or services that: (1) meet significant market needs in a way no one has previously tried; (2) reach a market of sufficient size that is of interest to investors or corporate players; (3) exude a vision that reaches beyond current products with the aim of creating a world-leading venture; and (4) require high levels of technical expertise and expense.

Furthermore, they highlighted four essential components for initial investor support that LS does not adequately address [26]. These include: (1) a significant market need and opportunity with a rapid growth potential that no one has previously tried; (2) an outstanding team who can execute; (3) a differentiated tech or business solution that beats the competition; and (4) a value proposition (and business plan) that defines the company’s value, strategy, and implementation (and attracts needed capital).

Interestingly, Netscape founder and Silicon Valley venture capitalist (Andreessen Horowitz), Marc Andreessen, shared a similar view [27]. In defining how entrepreneurs could run astray, he highlighted that not all startups could be lean startups. He added that sometimes, startups need to begin with an audacious idea or vision that one cannot do on a small scale first. He illustrated this point by highlighting the Macintosh and Elon Musk’s venture ideas. Andreesen admonishes the acceptance of “the pivot” and giving up to quickly, whereas the entrepreneur could learn from the failure and use the lessons learned to guide one down the path of success.

Marketing, Sales, and Revenue Growth Gaps

Heitmann highlighted that LS approaches the business from more of an engineering perspective rather than from a sales and marketing vantage [10]. He observed that having an engineering and experimentation view solely carries the risk of running out of money.

To this point, he added that the entrepreneur needs to shift quickly to sales and marketing, as these functions are closest to the customer and to turning the idea into actual revenues. This point was critical because the startup needs to generate cash flow to sustain itself and its investors.

Mougayar, a Toronto-based entrepreneur and venture advisor, reinforced Heitman’s point regarding shifting to sales and marketing quickly, as he cautioned entrepreneurs not to use LS as a crutch [28]. He observed that LS is not a growth methodology, but preferably one to gain P/MF quickly and cost-effectively. While he advocated that startups use LS, he recommended that entrepreneurs should transition quickly to focus on growing the venture. He explained that many entrepreneurs did not have much sales and marketing experience. To this end, he added that they do not pursue these key activities with much vigor or effectiveness. Hence, Mougayar noted that LS could be used as a crutch for these entrepreneurs not to go out and do all that is necessary to engage the marketplace to generate revenue. He continued that these individuals continue to pivot (and waste time) in trying to make a better mousetrap.

Andreessen also voiced his critique of LS as related to sales and marketing [27]. He observed that those are using lean use it as an excuse to spend on marketing and sales, two functions critical for growing a business that some startups might fail to the full appreciate or use in the early stages of the venture. He cautioned entrepreneurs about not letting lean become a crutch to their growth [27,28]. To this point, he added that for a company to change the world, sales and marketing, and, thus, capital, would be needed to hit the market with force to disrupt entrenched competitors or reliance on the status quo.

Conclusions

This review identifies several areas in which evidence, experience, and professional opinion highlight issues and learnings concerning the limitations of LS (Figure 2). The first is that practitioner experiences with LS are inconsistent. Furthermore, most do not reflect successful implementation. The inconsistency of experiences can be due to lack of understanding, mixed interpretations on how to use, type of industry, cultural considerations, and proper (and consistent) practice of the methods. The second involves methodological issues such as setting up proper hypotheses and experiments, using a proper MVP, and pivoting. A related consideration involves customer discovery and interviewing practices in which interviewer biases and challenges in getting the right customers to engage do exist. Thirdly, LS may not translate to a “breakthrough” venture and does not provide for all the essential capabilities for success the venture capitalists seek in a startup. Finally, it lacks a market and sales emphasis, which is essential to acquiring customers, generating revenue, and fostering growth.

These learnings prod us to consider what entrepreneurs need to know in moving forward with LS. They should realize that LS is one methodology to draw on from their entrepreneurial “toolkit,” but it does not adequately provide for all essential capabilities for investor support and startup success. When using LS, they should recognize these limitations and guard against individual tendencies and biases that will limit their effectiveness in using the methodology. Furthermore, they should engage in training and mentorship to fully understand and use the framework, the skills needed, and the processes to correctly (and consistently) implement the methodology.

Further research should be conducted to build on these findings and explore new avenues. Such work could examine several areas to characterize application and practice. Potential topic areas to explore include: (1) best practices; (2) outcomes beyond performance in a pitch competition; (3) gaps in implementation; (4) strategies to ensure consistency in the implementation of LS practices; (5) ideal domains where LS would successfully apply; and (6) how to use LS with other methodologies. Furthermore, research into the underpinnings concerning the role that LS (and its limitations) play in developing dynamic capabilities of a startup firm, entrepreneurial intent, entrepreneurial orientation, and entrepreneurial self-efficacy, learning behavior, and planned behavior may also serve as a useful contribution to entrepreneurial theory and practice.

Acknowledgement

The authors would like to thank Phillip T. Powell, PhD, Associate Dean of Academic Programs and Associate Clinical Professor of Business Economics at the Kelley School of Business at Indiana University, for his review of this manuscript and insightful comments.

Note: Reprinted from [29] Arch Bus Adm Manag: ABAM-104. DOI: 10.29011/ABAM-104. 100004 with permission from Gavin Publishing.

Figure 1: Blank’s Steps to Lean Startup (Blank 2013; York 2018).

Figure 2: Lean Startup Limitations Overlaying Essential Steps of the Methodology [13,29].

References

1. Blank S, Dorf B (2012) The Startup Owner’s Manual. Pescadero, CA: K&S Ranch, Inc.
2. National I-Corps™ grants. NSF I-Corps Sites Program (I-Corps Sites) Program Solicitation NSF 14-547.
3. Blank S (2013 [a]) Why the Lean Start-Up Changes Everything? Harvard Business Review.
4. Blank S (2013 [b]) The Four Steps to Epiphany-Successful Strategies for Products That Win. Pescadero, CA: K&S Ranch, Inc.
5. Ries E (2011) The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses. New York, NY. Crown Business (Division of Random House, Inc).
6. Teece DJ, Pisano G, Shuen A (1997) Dynamic Capabilities and Strategic Management. Strategic Management Journal 18: 509-533.
7. Frederiksen DL, Brem A (2017) How do entrepreneurs think they create value? A scientific reflection of Eric Ries’ Lean Startup approach. Int Entrep Manag J 13: 169.
8. Ladd T (2016) The Limits of the Lean Startup Method. Harvard Business Review.
9. Ladd T (2016) Business Model Validation as an Entrepreneurial Dynamic Capability. Academy of Management Annual Meeting. Anaheim, CA (Peer-reviewed).
10. Heitmann J (2014) The Lean Startup. A Pragmatic View on Its Flaws and Pitfalls. MA Thesis. The University of Twente.
11. Blomberg A (2012) The Lean Startup Approach and Its Applicably Outside of Silicon Valley. Master’s Thesis. Copenhagen Business School.
12. Gustafsson A, Qvillberg J (2012) Implementing Lean Startup Methodology- An Evaluation. Master’s Thesis at the Chalmers University of Technology.
13. Jureen K (2014) The Lean Healthcare Entrepreneur. Master’s Thesis. KTH. Stockholm, Sweden.
14. Nilsen JT, Ramm NA (2015) The Lean Startup: A Success Factor? A Quantitative Study of How Use of the Lean Startup Framework Affects the Success of Norwegian High-Tech Startups. Master’s Thesis. The University of Oslo.
15. Nirwan MD, Dhewanto W (2015) Barriers in Implementing the Lean Startup Methodology in Indonesia - Case Study of B2B Startup. DataProcedia - Social and Behavioral Sciences 169: 23-30.
16. Shafer H (2014) Innovation Games: A Lean Startup Secret Weapon Presentation. Innovation Games Summit.
17. Ng G (2014) Five Pitfalls of Running Lean Startup Experiments. Forbes.
18. Rao AS (2014) Minimum Viable Product (MVP) for Product startup: An Indian perspective. Presented in Forum for Knowledge Sharing IX Annual Conference. Conference Theme: Technology: Corporate and Social Dimensions, Venue-National Institute of Advanced Studies (NIAS), Bangalore.
19. Kortmann P (2012) The Problem with a Lean Startup: The Minimum Viable Product.
20. York JL, Danes JE (2014) Customer Development, Innovation, and Decision-Making Biases in the Lean Startup. Journal of Small Business Strategy 24: 21-39.
21. Kahneman D (2011) Thinking, Fast, and Slow. New York. Macmillan.
22. Soll JB, Milkman KL, Payne JW (2015) Outsmart your own biases. Harvard Business Review 93: 65-71.
23. Stanovich KE, West RF (2004) Individual differences in reasoning: Implications for the rationality debate. Behavioral & Brain Sciences 23: 645-665.
24. Holcomb TR, Ireland RD, Holmes RM, Hitt MA (2009) The Architecture of Entrepreneurial Learning: Exploring the Link among Heuristics, Knowledge, and Action. Entrepreneurship Theory and Practice 33: 167-192.
25. Chen T, Simon M, Kim J, Poploskie B (2015) Out of the Building, Into the Fire: An Analysis of Cognitive Biases During Entrepreneurial Interviews. New England Journal of Entrepreneurship 18: 59-76.
26. Kressel H, Winarsky N (2015) Why the Fail-Fast Approach Is Not Right for Breakthrough Ventures. Harvard Business Review.
27. Kern E (2015) Marc Andreessen: Not Every Startup Should Be A Lean Startup or Embrace the Pivot.
28. Mougayar W (2013) Do Not Let Lean Startup Become a Crutch. Startup Management.
29. York JM (2018) Putting Lean Startup into Perspective: A Novel Approach for Discovering and Developing a Successful Business Model. Arch Bus Adm Manag. DOI: 10.29011/ABAM-104. 100004.