Lean Startup Best Practices: Write hypotheses you can learn from.

~ James Birchler ~ 2 Comments

If you practice Lean Startup methodologies, you’re likely familiar with Build-Measure-Learn. It seems like the directive here is to first build something quickly, then  measure some results, and then try to learn, all as quickly as possible. The problem is that too many people have taken this literally, building without first pausing to articulate what they think they know about their business and customer, and what it is they want to learn next.

What we actually need to do is Learn-Build-Measure. In other words, think first about what you already know about your business and customer, then form a testable hypothesis to validate, and then move on to building and measuring.

Now you might be thinking, “But it’s a loop, so why should it matter where we start?”

It matters because building without some reasonable understanding of your business and customer is just guessing. And if you’re guessing, you’re in the realm of superstition rather than science.

Building without an understanding of your customer is superstition, not science. – Tweet This

Unless you’ve put some serious thought into testing hypotheses, running experiments, and applying Lean Startup methodologies, it’s likely that you’ve been guessing about the results of your experiments rather than learning from them. Every time this happens, you lose a valuable chance to validate and improve your understanding of your business and customers, and you waste time and money.

Writing a good hypothesis is hard, but it’s worth the effort. A good hypothesis makes it easy to design an experiment to test and learn from it—by making clear what it is you are validating or invalidating—usually your current understanding of your business and customers.

A proper hypothesis is an educated guess—just like we learned in our science classes growing up. What I see most often, though, are merely guesses: spaghetti on the wall and shots in the dark. How could this be happening, with all of our understanding of Lean Startup, and of science?

The Lean Startup Build-Measure-Learn loop is really an application of the scientific method. Check it out (slide 24 from that presentation, which touches on several Lean Startup practices):

lean startup loop is the scientific method

This means that we need to use care to ensure we take our learnings and shape them into new ideas and generate new hypotheses.

tweet-thisThe Lean Startup Build-Measure-Learn loop is an application of the scientific method. – Tweet This

Many people are unable to clearly articulate their current understanding and assumptions about their business and their customer. If you are unable to clearly articulate these and you are trying to run experiments, it’s a red flag. It’s probably better to start with developing a better understanding of your customer than running haphazard experiments.

Here is a template to use for crafting hypotheses that you can learn from, and hopefully help clarify whether running an experiment is the next right step.  If you can write a proper hypothesis, you’re probably ready to run an experiment. I’ll start with this simple version:

Because we believe X, if we do Y, we expect Z to happen.

Writing a hypothesis with details about your business requires a solid understanding of our business and customer.  We must assert something about what we know, or have already learned that we can then validate. This is true validated learning.

Let’s add more detail using a lemonade stand business as an example and get one step closer to having a hypothesis we can validate:

Because we know our customers prefer colder lemonade in warmer weather, if we add ice to each cup of lemonade we sell, we expect higher customer satisfaction and more sales.

I like to go even further, adding an assessment of the potential outcomes to the experiment plan. This gives you a better idea of why you’re running the experiment in the first place, forces you to think about how you’ll use what you learn, and importantly, forces you to consider some of your assumptions:

If we are correct, we would also expect that customer surveys would also validate a preference for ice, that we might be able to run tests to find the quantity of ice that maximizes sales, and that our Net Promoter Score would increase.  If we are wrong, it might indicate that customer preference for cold lemonade might be influenced by other factors such as the precise ratio of ice to lemonade, the exact temperature outside, the quantity of sugar, lemon juice, and water in our recipe, or the price of a drink from your competitor down the street.

This assessment is crucial, and perhaps the hardest part of running an experiment as it reflects the thinking behind the experiment. Good experiments require thoughtful planning up front, even when practicing Lean Startup methodologies.

tweet-graphic-transGood experiments require thoughtful design, even for a lean startup. – Tweet This

Even our simple example exposes many factors that might affect experiment design and results interpretation. Taking time to think, articulate what you know, and expose your assumptions while forming a proper, testable hypothesis will help you do truly validated learning, and help you to fulfill the promise of the Lean Startup.

Protect Your Engineering Culture As You Scale: Write Down Your Team’s Cultural Manifesto

~ James Birchler ~ Leave a comment

Culture eats strategy for breakfast. —Peter Drucker   Tweet this.

I frequently speak with startup founders and leaders who are growing and scaling their teams, and among the most common questions I hear are about building and maintaining engineering team culture. This is because that while establishing the culture you want at your startup is hard, maintaining it as you scale can be even harder. I’d like to share a few of the lessons I’ve learned, some of them the hard way, during my last 15 years working with engineering teams.

Scaling your team can wreak havoc on your established culture, overwhelming it with surprising speed as new hires bring fresh ideas and expectations about how your teams function and collaborate — but only if you let it.

In other words, the real threat to your culture is not your newcomers, it is the challenge of being mindful and selective about what you decide to incorporate.

A written manifesto is a simple and effective tool to manage the evolution of your culture to maintain its core attributes.

Culture is the manifestation of the customs, institutions, attitudes, behaviors, and achievements of a group, regarded collectively. In other words, culture is both a product and a process. Your culture evolves—either maintaining or changing—every day in the myriad ways you and your team interact, converse, and collaborate, literally manifesting your culture and making it obvious to all. Your culture is a living archive of all your team has done together.

Your culture is a living archive of all your team has done together. Tweet this.

When you hire new people to scale your team, they begin with none of the rich background of your culture that you and your team have created, lived, and made manifest over time. Unless you create a way for your new team members to access your archive of cultural history, they are going to have a hard time maintaining the culture you built.

By writing down your team’s manifesto—a public declaration of your culture and the standards by which you agree to practice, operate, and be held accountable—you are creating a concrete artifact. Your manifesto if powerful because it goes beyond ephemeral words and actions: it becomes a referent for your team members, both new and old, to use as a waypoint as they navigate their daily choices of words and deeds which when combined, either reinforce or undermine your culture.

I recommend a basic process framework for creating written artifacts which allows for input from all quarters to ensure buy in, while avoiding the paralysis of “design by committee.” Start with ideas, build to a proposal, and finally present a recommendation for adoption.

It is important to communicate this overall process to your team to set expectations and ensure everyone understands when and how they will have input. The goal here is to have your team bought in, and it is important that people feel like they have a voice in the process.

When you’re ready to write your manifesto, start with a core group of leaders and ask them author the manifesto with you. Begin with an idea brainstorm, and ask them to begin drafting a proposal. As they share versions of the manifesto, ask them to incorporate feedback from others in a way that authentically captures the feedback. This is challenging work, as they will need to balance a variety of perspectives. Do not rush this process, and expect this may take several weeks from start to finish.

Make your first shares to a small but diverse set of individuals who can quickly provide feedback to ensure your proposal is directionally correct. From there, share it with your entire team, inviting input, comments, and feedback. Share new versions of the revised manifesto periodically rather than in real time to allow for the team drafting the document to think through, discuss, and carefully craft their revisions.

Here are some sample statements from real engineering team manifestos. You’ll see they cover a wide range of issues around how my teams have chosen to work together, from the practice of software engineering to mindset and communication:

We develop and test our code in a safe and controlled manner using modern tools and processes. We are careful to isolate potential instability in shared environments, services, and data. We consider and plan for potential areas of risk, mitigating them whenever possible.

For any project, we always determine how we will measure its success or failure, and how we will respond to either of these outcomes, and we commit to that follow-up work.

We work iteratively, shipping products quickly to test our riskiest assumptions first to validate our ideas and learn quickly. We practice the 80/20 rule, and believe that rapid iteration, “Launch and Learn,” is a better strategy to deliver value to our customers than seeking perfection before shipping.

We mix modern best practices with pragmatic choices to craft software we are proud of.

Every bug we fix or new feature we build has automated test coverage. If it’s hard to write tests for something, we figure out why and fix it so that it won’t be hard in the future. Code without tests is our “nuclear option” – a last resort and a decision that we make reluctantly and rarely.

We leave each piece of code we touch better than we found it. We refactor and clean up code, but we’re careful to prioritize that work based on the value it delivers to customers, or whether it makes it easier to change the code in the future.

We actively support one another. We create safe spaces for communication so we can feel safe being vulnerable to speak candidly, and ask for help when we need it. We want it to feel comfortable to both seek and offer help.

We take the time to reflect on, learn from, and talk candidly about both our successes and our mistakes. We practice retrospectives and debriefs to capture learnings so we can repeat the good and discontinue the bad. We assume positive intent, and seek understanding not blame.

We use processes as guides, but do not follow them blindly. We expect our processes to change as our context and awareness changes, and as we learn and grow.

Once your team has a version of the manifesto they feel represents your culture and the standards by which the team agrees to practice, operate by, and be held accountable to, ask them to present it as a formal recommendation to the approver (usually the VPE or CTO for an engineering team).

At this point, everyone on the team ought to be familiar with the manifesto, and when you ask they should tell you that it resonates strongly with them. If this is not the case, send the recommendation back for more revisions.

If it is ready, it’s time to roll it out. Plan to communicate your manifesto in multiple venues and using every possible medium. Add a copy to your engineering department wiki or documentation, and provide pointers everywhere your team hangs out online: pin it to your team Slack channel, email to your team list, post in IRC — and in all cases make sure it’s archived and searchable. Make posters and print copies of your manifesto and display them prominently.

Next, hold a ceremony at your next engineering team allhands to announce your manifesto and celebrate it — make it real. Announce it at the company allhands, too, ensure that everyone knows how your team plans to hold itself accountable to delivering for customers and the business. Ensure your managers know to revisit your manifesto regularly, until it too is part of your team’s regular practice of engineering and its culture.

Image Credit: Image cropped from “science” by Robert Couse-Barker, Creative Commons License.

Lean Startup Best Practices: Build a culture that supports learning.

~ James Birchler ~ Leave a comment

Building valuable products for customers is challenging, and Lean Startup methodologies can help. The very act of learning itself is fundamental to these principles, and it’s worth attending to: one of the core tenets of Lean Startup practice is the Build-Measure-Learn loop, an application of the scientific method.  But how do you maximize your learning with every pass through the loop, in the face of both success and inevitable failure? A healthy culture that supports learning can help ensure that you gain maximum value from every experiment you run.

It’s easy when your data comes back validating your hypotheses about your customer and business, but what happens when you upset your beliefs about your customer or business?I’ve seen too many teams ignore hard data and learnings only to maintaining a course toward product development failure.  And even science-based Lean Startup methods will fail without a culture that supports learning. To illustrate why, I’ll briefly draw your attention to an example from the history of science.

Copernicus, Giordano Bruno, and even Galileo, widely known as a founder of modern science, were persecuted, arrested, and even burned at the stake, in part because the prevailing culture didn’t support their scientific methods of learning and the data and findings they presented.

The Catholic Church began the Inquisition in 12th-century France to combat heresy—but it took its toll on science throughout the 16th and 17th centuries before it was finally abolished in the early 19th century.

inquisition

A product manager presenting customer data to the executive team, or two priests showing the application of torture under the supervision of the Inquisition?

Nicolaus Copernicus formulated the heliocentric model of cosmology, placing the Sun at the center of the universe rather than the Earth. This was literally Earth shattering news at the time for the Catholic Church, which saw this as a threat to their authority  based in their belief system—which included the Earth being at the center of things.

Instead of dealing with the facts of the situation, they attacked the messengers. Copernicus so feared the Inquisition that he delayed publishing his findings until he was on his deathbed in 1543. Giordano Bruno followed Copernicus and was burned at the stake. Galileo was placed under house arrest until his death.

Why was is so hard for these scientists to share their data and findings? The answer is complicated, but many product development professionals know the answer all too well. Suffice it to say that the prevailing culture espoused by the leadership at the time did not support their findings, particularly when that learning contradicted the prevailing belief system.

The parallels with modern product development teams, even or perhaps especially those practicing Lean Startup principles, are stark.

What is the culture like in your company? Is it hard to speak up when you have learnings to share and the news seems to contradict your quarterly goals or yearly plan? Does the highest-paid person’s opinion matter more than your customer data? Do teams and leaders prefer to stay the course, even when data is signaling a change might be warranted? I’m aware of too many companies that suffer from a combination of hubris, denial of reality, and delusions that everything is going according to plan—even as the projects, teams, and in the worst case, the entire company, is headed toward failure.

So what can we do?

The point I’m making is that culture matters. Creating a culture that supports continuous learning can help protect you in the face of visionary leaders with compelling reality distortion fields or your own subconscious desire to believe that your project is on track, even when it’s not.

Creating a safe space to learn within your company—a culture that supports learning in the face of both success and failure—while challenging, is worth the effort. Even with the world’s best product team  building, experimenting, and measuring results, Lean Startup methodologies won’t work without a culture that supports learning from your data.

I’ve discussed previously why it’s often better to reorder the Build-Measure-Learn loop to start with learning rather than building, as building without an understanding of your customer is superstition, not science. Learning literally comes first, beginning with what you already know about your customers and your business. Good hypotheses are also required, as they drive experiments that you truly can learn from.

But what of culture? Culture is at once a process and a product. This means we have the power to change culture, but it requires conscious, meaningful effort. Think about how you want your culture to be, then take action. Everyone on your team contributes in minute ways to building, changing, and refreshing your culture with every interaction and every word spoken.

Culture is at once both a process and a product; we can make change with conscious, meaningful effort. —Tweet This.

Leaders have a responsibility here, of course. Culture starts at the top and permeates your organization. Your leadership team must be aligned and supportive—but that is not enough.

Leaders must make conscious efforts to build and maintain a culture of learning. Candor, based in vulnerability and accountability, is required. Clear communication about how the culture is meant to be will help–it should be written down, easily accessible, and referenced often in your practice.

Leaders, like everyone else, need to make an ongoing, conscious effort to build and maintain a culture that supports learning. Practice is key, and leading by example works. Executives can and should conduct root cause analysis when things don’t go as expected, and discuss the learnings and action items openly within the company.

It’s won’t be enough that one of your executive, engineering, product, or other teams practice a learning culture in isolation, so plan ways to ensure everyone practices together. Most teams are multidisciplinary, so ensure that everyone participates in learning moments through reporting on experiments, and during project retrospectives and postmortems. Learnings and action items should be communicated widely, and used as inputs to your product development and planning processes.

Building and maintaining a healthy culture that supports learning will help you maximize value from every turn through the Learn-Build-Measure loop, and help you make good on the promise of the Lean Startup.

IMVU’s Agile Engineering Process — Interview with Clinton Keith of Agile Game Development

~ James Birchler ~ Leave a comment

I spent a day hosting Clinton Keith in the IMVU offices during the Game Developer’s Conference in March, and I spent time describing IMVU’s product development process and our implementation of Agile and Lean Startup methodologies. He heard a lot of interest in our success from people at GDC he spoke to about IMVU, and he asked if I would do a Q&A session. Here is an excerpt of our interview posted to his Agile Game Development Blog:

Agile Game Interview – Agile Engineering for Online Communities 

James Birchler is an Engineering Director at IMVU, where he implemented and manages IMVU’s product development process using Agile and Lean Startup methodologies. Prior to joining IMVU in 2006, he redesigned the product development process at Bargain Network, Inc., helping complete the company’s acquisition by Vertrue, Inc. Previously, James was a Director at There.com, responsible for technical design, creative design, and content of the web applications and GUI for There’s virtual world application. James will be presenting on Lean Startup methodologies at the Startup Lessons Learned Conference on April 23 in San Francisco, and is a frequent contributor to the IMVU Engineering Blog.

Clinton Keith: What is IMVU?

James Birchler: IMVU, Inc. (www.imvu.com) is an online community where members use 3D avatars to meet new people, chat, create and play with their friends. IMVU has reached 40 million registered users, 10 million unique visitors per month and a $30+ million annualized revenue run rate. IMVU has the world’s largest virtual goods catalog of more than 3 million items, almost all of which are created by its own members.  IMVU achieved profitability in 2009 and is hiring aggressively with plans to double the size of its team in 2010.

CK: What is the overview of the IMVU engineering process?

JB: Our engineering team practices what people refer to as “Agile” or “Lean Startup” methodologies, including Test-Driven Development (TDD), pair programming, collective code ownership, and continuous integration. We refactor code relentlessly, and try to ship code in small batches. And we’ve taken continuous integration a step further: every time we check in server side code, we push the changes to our production servers immediately after the code passes our automated tests. In practice, this means we push code live to our production web servers 35-50 times per day. Taken together with our A/B split-testing framework, which we use to collect data to inform our product development decisions, we have the ability to quickly see and test the effects of new features live in production. We also practice “5 Whys” root cause analysis when something goes wrong, to avoid making the same mistakes twice.

CK: How do you get so many changes out in front of customers without causing problems, either for your customers or your production web servers?

JB: I think it’s important to point out that sometimes we actually do introduce regressions and new bugs that impact our customers.  However, we try to strike a balance between minimizing negative customer impacts and maximizing our ability to innovate and test new features with real customers as early as possible. We have several mechanisms we use to help us do that, and to control how customers experience our changes. It boils down to automation on one hand, and our QA engineers on the other.

First the automation: we take TDD very seriously, and it’s an important part of our engineering culture. We try to write tests for all of our code, and when we fix bugs, we write tests to ensure they don’t regress. Next, we built our code deployment process to include what we call our “cluster immune system,” which monitors and alerts on statistically significant changes in dozens of key error rates and business metrics. If a problem is detected, the code is automatically rolled back and our engineering and operations teams are notified. Next, we have the ability to limit rollout of a feature to one or more groups of customers–so we can expose new features to only QA or Admin users, or ad-hoc customer sets. We also built an incremental rollout function that allows us to slowly increase exposure to customers while we monitor both technical and business metrics to ensure there are no big problems with how the features work in production. Finally, we build in a “kill switch” to most of our applications, so that if any problems occur later, for example, scaling issues, we have fine-grained control to turn off problematic features while we fix them.

Read the rest of Agile Game Interview – Agile Engineering for Online Communities …