Lean Experiments: From observation to experiment

There is a bunch of literature and reading out there about lean experiments, and the lean startup methodology. So by way of disclosure this by no means is something I have created from scratch. What follows is a synthesis of a couple of workshops that I conducted in 2016 for different audiences. In the course of the workshops, I created a 'canvas' (because they're so in vogue) to help someone think through the process of running an experiment as part of the work that they do. The goal of sharing this as a resource is to see if others find it useful, for more people to use and test it and most importantly to point out how this could be made better.

What are lean experiments?

My first exposure to the method of lean experiments (though at the time I didn't realise that that's what it was) was a research and prototyping exercise that was conducted by a friend as part of his graduation project. His team was working with a global printing equipments manufacturer who leases out plotters to local print shops across India, and in their case Ahmedabad. Their goal was to try and design a 'portal' where local architects, designers and businesses could post files for printing that would be queued at the closest print shop, produced and delivered.

In an attempt to understand how they might design the functionality of this platform, they substituted the platform for a manual 'portal'. They partnered with a couple of architecture studios in the city and a handful of printers. The idea was that the studio would email them the file for print, and the team would mimic the functionality of the portal, that is getting the print produced, and delivering to the person who ordered the print. Through the application of this method, they were able to over a short duration understand how they might design a lot of the details of the 'portal', besides being able to avoid some mistakes that were bound to happen if they had tried building a full fledged portal to begin with.

Lean experiments became an integral part of my vocabulary and toolkit through the able mentoring by Villgro while working on Amrutdhara. Early on in the process, we were trying to get them to fund a full fledged water kiosk that would cost somewhere in the ballpark of INR 300,000 or USD 6700. At that point in our evolution, we didn't yet have answers to fundamental questions about the proclivity of potential consumers to consumer or buy loose water for consumption. They encouraged us to take a much smaller amount, about INR 10,000, to do run a lean experiment to answer that question. To that end, we ran some small experiments by setting up a basic (less than legal) table and umbrella in a few locations in Chennai and Pondicherry and started hawking loose water.

Essentially lean experiments by my definition are quick, simple, cheap, short experiments that you run to find answers to fundamental questions you might have about the work that you're doing, or to test the assumptions in an idea that you have.

Thinking through a lean experiment

The first step to running an experiment is to have a clearly articulated hypothesis, or question or assumption that you want to test. In the Amrutdhara case, we wanted to know whether people would buy loose water, and what their perceptions around a service/product like this would be. A simple structure that might help in your work is to think about it in a few steps, from observation to testable hypothesis.

So on the canvas, start with a single observation or a set of similar observations. What is it that you understand from this? What is the insight? What is it that you can glean from those observations? Finally, what is a possible solution to the problem? And if you were to execute the solution what do you hope to observe?

So for the Amrutdhara case, the observation was that people in public spaces need drinking water, but currently they don't have any option but to use water packaged in disposable plastic bottles. The insight was around the quality consciousness of the consumer, and that if their need for quality was satisfied they might buy loose water as well, consuming it in a tumbler or their own bottles. Finally the solution was to offer quality assured drinking water in public spaces, and the hypothesis was that when assured of quality consumers would happily purchase and consume loose water rather than more expensive packaged water.

The next step is to design the lean experiment. And here the activity is key, but there are three elements that are critical to thinking the lean experiment through, and to ensure success in gaining insight into the hypothesis you're testing.These are:
Choice: It is critical that you force the consumer to make a choice in which they part with something of value, be it money, time or energy. In the first case with the printing portal, the architecture firm took the risk of losing money on bad prints, and with the Amrutdhara case it was quite simply money.
Measurement: What are you measuring? And what does the measurement mean? What are your learnings? In the case of the Amrutdhara case, we learned a bunch both from those who bought water and equally (if not more) from those who didn't. The key is not too just measure quantitatively, but to engage with consumers to understand why they made the choice that they did.
Documentation: While this is fairly self-explanatory, I encourage you to make notes while planning the experiment on how and what will be documented, and importantly who will do the documentation. While this might seem somewhat pedantic, documentation as an afterthought leads to sub-par results while analysing the results of the experiment.

Done With The Experiment, Now What?

Let's assume that you're done with the experiment, and things went off exactly as planned, you documented everything well. The next step then is to analyse the findings, and the key being not to bias your analysis towards proving what an amazing success things were. Being honest with yourself is absolutely imperative. At the end of the analysis, ask yourself, 'do I know enough?'. In the case of Amrutdhara, we felt like we didn't so we repeated the experiment in a couple of different locations, sometimes switching to different sides of the road with interestingly different results.

There's two ways to go forward. If you don't know enough, go back a couple of steps, to your new observations and iterate the experiments and repeat till you have clarity (or you run out of money or patience). If you think you do know enough, depending on your context, you might want to build in more fidelity or go to scale. With Amrutdhara, based on our learnings we pivoted the entire company from standalone water kiosks to modular water ATMs that would be in every small shop, and our next set of lean experiments was with a simple prototype.

So there you have it, a quick primer on running a lean experiment. I'd like to share this canvas as a kind of guide for running your own lean experiment. And if you do, please share how it went!