Test-and-Learn is the new mantra in the tech world. Be it new product features, new policy ideas, marketing messaging, or email content, test-and-learn is gaining steam, graduating from simple A/B tests to complex multi-variate tests. However, when it comes to fundamentally understanding consumer behavior and preference this approach has not lived up to its promise in the industry. The primary reason for this is that due the very nature of this approach, you have to develop hypotheses upfront. Even the most diligently designed test will give you the best option among the ones you tested. It is incapable of telling you if you missed a great idea.
So developing the right hypotheses to test is key. To understand how to develop creative, effective hypotheses we must first be clear on why we are testing in the first place.
Three Reasons For Testing
1. We are not our customers. A senior executive at a large payments company asked a group of about 100 employees a simple question — if you were at a physical store buying something for less than $50, how many of you would prefer to pay using cash. About 10% hands went up. Then he went on to share a new market research that suggests 80% of American consumers (potential customers of this payments company) would prefer to pay using cash for offline transactions of less than $50.
This huge gap in the fundamental consumer mindset runs home the point that we are not our customers, i.e., we are not a representative sample of the base that is our customer.
We must let go of the belief that we can sit in a conference room with decades of collective experience among us, and dream up what customer would want, like, and prefer. So if we cannot trust ourselves to know what customers want, why not ask them?
2. Customers are unable to articulate what they want. Customers are particularly poor at articulating what they want, primarily because they don’t know what is possible and what isn’t. This is the reason why customer-focused companies such as Proctor and Gamble and Intuit like to passively observe customers using their products. It gives them invaluable insights for improving their products and services.
A large producer of consumer goods released a new laundry detergent in a country in Latin America. Sales were not doing good despite big advertising budget and objective evidence of superiority of their product over the competition. They followed their customers home to observe how their product was being used. It was nothing short of a revelation! This was a third world country where women washing clothes had to fill buckets of water and bring it to the place of where clothes were to be washed. The company ended up creating a “high-efficiency” detergent that required half the amount of water as before. This reduced the labor of these women by half, and the product was an instant success!
There is a great talk by Malcolm Gladwell illustrating the same point in the context of spaghetti sauce.
There is no way these women could have asked for a high-efficiency detergent because they did not know it was a possibility. Nor could the customers that Gladwell talks about have known that what they really wanted was extra chunky sauce. Customers are often unable to articulate what they want primarily because they don’t know whats possible and they have not experienced it yet.
So, if we cannot predict what customers want, and customers are unable to articulate what they want, what is the solution? Why not create a few options and ask them to choose? Here in lies the holy grail of experimentation — developing the right hypotheses to test, ensuring that one of the ideas we are testing is the “magic answer”. Development of creative and effective hypotheses relies on our understanding of consumer behavior — the understanding that response of people is often irrational, but that there is a method to the madness.
This brings us to the third reason why we need to test in the first place.
3. Customers may respond to our offers and propositions irrationally. The field of behavioral economics sheds ample light on how irrational the response of most people is when faced with certain kinds of situations. While traditional economic theory (based on the homo economicus being) says that all parties will act in their own economic interest and the invisible hand of the free market will take care of the rest, behavioral economics (based on decades of social science research) shows that people often make economically sub-optimal decisions based on emotions and irrational considerations. Being keenly aware of this body of knowledge allows you to develop hypotheses (and treatments addressing them) that are effective and have a high probability of producing a disproportionate impact.
Whats Coming: In the next post I will discuss some principles of behavioral economics that highlight the irrationality of decisions, illustrate the contrast between a typical consumer and the homo-economicus being, and yet impress upon you that it is all predictable.
Questions for readers: Do you have examples/experiences relevant to this discussion? Have you observed something similar to the “laundry detergent” example above, where customers did not know what they really wanted.