A Test Is Only As Good As The Hypothesis It Tests

Test-and-Learn is the new mantra in the tech world.  Be it new product features, new policy ideas, marketing messaging, or email content, test-and-learn is gaining steam, graduating from simple A/B tests to complex multi-variate tests. However, when it comes to fundamentally understanding consumer behavior and preference this approach has not lived up to its promise in the industry. The primary reason for this is that due the very nature of this approach, you have to develop hypotheses upfront. Even the most diligently designed test will give you the best option among the ones you tested. It is incapable of telling you if you missed a great idea.

So developing the right hypotheses to test is key. To understand how to develop creative, effective hypotheses we must first be clear on why we are testing in the first place.

Three Reasons For Testing

1. We are not our customers. A senior executive at a large payments company asked a group of about 100 employees a simple question — if you were at a physical store buying something for less than $50, how many of you would prefer to pay using cash. About 10% hands went up. Then he went on to share a new market research that suggests 80% of American consumers (potential customers of this payments company) would prefer to pay using cash for offline transactions of less than $50.

This huge gap in the fundamental consumer mindset runs home the point that we are not our customers, i.e., we are not a representative sample of the base that is our customer.

We must let go of the belief that we can sit in a conference room with decades of collective experience among us, and dream up what customer would want, like, and prefer. So if we cannot trust ourselves to know what customers want, why not ask them?

2. Customers are unable to articulate what they want. Customers are particularly poor at articulating what they want, primarily because they don’t know what is possible and what isn’t. This is the reason why customer-focused companies such as Proctor and Gamble and Intuit like to passively observe customers using their products. It gives them invaluable insights for improving their products and services.

A large producer of consumer goods released a new laundry detergent in a country in Latin America. Sales were not doing good despite big advertising budget and objective evidence of superiority of their product over the competition. They followed their customers home to observe how their product was being used. It was nothing short of a revelation! This was a third world country where women washing clothes had to fill buckets of water and bring it to the place of where clothes were to be washed. The company ended up creating a “high-efficiency” detergent that required half the amount of water as before. This reduced the labor of these women by half, and the product was an instant success!

There is a great talk by Malcolm Gladwell illustrating the same point in the context of spaghetti sauce.

There is no way these women could have asked for a high-efficiency detergent because they did not know it was a possibility. Nor could the customers that Gladwell talks about have known that what they really wanted was extra chunky sauce. Customers are often unable to articulate what they want primarily because they don’t know whats possible and they have not experienced it yet.

So, if we cannot predict what customers want, and customers are unable to articulate what they want, what is the solution? Why not create a few options and ask them to choose? Here in lies the holy grail of experimentation — developing the right hypotheses to test, ensuring that one of the ideas we are testing is the “magic answer”. Development of creative and effective hypotheses relies on our understanding of consumer behavior — the understanding that response of people is often irrational, but that there is a method to the madness.

This brings us to the third reason why we need to test in the first place.

3. Customers may respond to our offers and propositions irrationally. The field of behavioral economics sheds ample light on how irrational the response of most people is when faced with certain kinds of situations. While traditional economic theory (based on the homo economicus being) says that all parties will act in their own economic interest and the invisible hand of the free market will take care of the rest, behavioral economics (based on decades of social science research) shows that people often make economically sub-optimal decisions based on emotions and irrational considerations. Being keenly aware of this body of knowledge allows you to develop hypotheses (and treatments addressing them) that are effective and have a high probability of producing a disproportionate impact.

Whats Coming: In the next post I will discuss some principles of behavioral economics that highlight the irrationality of decisions, illustrate the contrast between a typical consumer and the homo-economicus being, and yet impress upon you that it is all predictable.

Questions for readers: Do you have examples/experiences relevant to this discussion? Have you observed something similar to the “laundry detergent” example above, where customers did not know what they really wanted.

This entry was posted in Uncategorized and tagged , , , , , , , , , . Bookmark the permalink.

8 Responses to A Test Is Only As Good As The Hypothesis It Tests

  1. Phil Young says:

    Really sound points. Someone once said: “a problem properly defined is half-way solved.” Applying this to your point, one could say that a properly stated hypothesis is (at least) half of an effective test.

    • mukulpatki says:

      Very well put Phil. And the key to developing good creative hypotheses is understanding how consumers behave, which is the topic of my next post, coming out in a day or so.

  2. sumitsehgal says:

    Mukul, this is great. Looking forward to your next post 🙂

  3. Saurabh says:

    Mukul – It is really well written post.

    I like the last two points; ‘ customers are unable to articulate what they want’ – that is they are not aware of possibilities of the way their ‘want’ can be fulfilled. Secondly, they ( we) all are irrational so it is not just ‘need’ but ‘want’ that decides our choices.
    Now, when we are applying psychology to it, the purpose is to make customers want something even if ( sometime) they don’t need it ( eg coke,pepsi I guess).

    Now my question is (just for discussion purpose ). lets say i want to launch a third drinks ‘pepke’ with almost same taste like coke or pepsi, how can i test whether it will work or not.

    • mukulpatki says:

      Thanks for your comment saurabh. you are right often companies first create a need and then provide a solution for it. However, i dont think it is about pushing products that customers dont need, because the ultimate choice is still in the hands of the consumer. Introducting new products such as “pepke” actually is an ideal use case for testing. The key again, though, is to discover what aspect of the product truly resonates with consumers, and accept it even if it does not align with your idea of the “value proposition” that your product provides.

  4. Sri Ravishankar says:

    Awesome Mukul, waiting for the next post 🙂

  5. Sanket says:

    Pardon my ignorance on the aspects of behavioural economics. From my experience in the service industry, I can say that, a visit to the ground is worth a thousand assumptions/hypotheses. The question remains, How representative is the representative sample? As per the law of averages, yes, a number of experiments may go totally wrong depending on the sincerity of effort, assessment of the target segment and of course a correct definition of the problem at hand.

  6. clifffy says:

    Very nicely written..
    Always heard the “think like your customer” quote…but afterall, its rarely possible..
    Wonder if there can be an acceptable framework for hypothesis creation in this scenarios..
    some check boxes/thumb rules which will make sure that the tests meet the objectives..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s