Consider the following...

​
  • In 2002, Danny Kahneman was awarded the Nobel Prize in Economics for his work on judgment and decision making, work that relied on a handful of 30-second experiments.
​
  • In 2007, the Obama campaign ran a small online experiment varying a simple webpage image and button phrase. The result netted the campaign over $50 million in donations.
 
  • In 2012, Microsoft ran a simple A/B test tweaking how ad headlines displayed in Bing’s search results. That small change netted Microsoft over $100 million in additional revenue.
 
These examples show how a single experiment can change the fortunes of the experimenter. But what exactly is an experiment?

What is an Experiment


​Experiments are a research method used to answer questions of cause and effect, i.e., does some “X” affect outcome “Y”. Whether you’re a social scientist, tech CEO, presidential candidate, or everyday person, you care about certain outcomes and what causes them. Does a specific treatment cure a disease? Will my new webpage attract more users? Do I look better with a hat than without? All of these questions concern cause and effect, and experiments can help us answer them.
 
Experiments are also known as “A/B tests” or “randomized controlled trials,” because every experiment should include:
 
  • A control condition to compare against the causal factor of interest, and
  • Random assignment of each participant to either the control or causal factor.
 
It may seem more complicated than a simple survey, but that’s because surveys and big data are better at describing and predicting outcomes than identifying whether some factor “X” actually causes an outcome.

Control - Why You Need It


Suppose you take a photo wearing a hat and survey your friends about how you look. With no comparison it’s hard to know what the data is telling you. If 60% say they like your outfit, that might seem good but you wouldn’t know that 80% would have said so without a hat. Yikes.
 
And there’s the question of what’s causing the lower ratings. Is it the hat? Your shirt? Your pants? It’s impossible to know. But surveying people with a similar “control” photo, one in which you’re not wearing a hat but keeping your shirt, pants, background, etc. the same gives us a comparison.

Randomization - Why You Need It


​Now imagine that you notice when you’re wearing a hat people tend to compliment you more than when you’re not wearing a hat. You may think the hat “causes” the compliments, but it could be that you wear a hat when you’re with a certain group of friends who just compliment more in general (reverse causality). Or maybe you only wear a hat when it’s sunny, and people are in a better mood when it’s sunny (omitted variable bias). In both cases there is a relationship between X and Y, but X does not cause Y. In fact, something else is causing X!
 
An experiment takes care of these issues by randomizing the “X” factor of interest (hat vs. no hat). Let a coin flip decide whether you wear a hat on certain days, and if you still get more compliments the hat is likely why, because now it isn’t the sun or your friends causing the hat. It’s the coin.

Conclusion


Experiments are great at answering questions of causality. This is why experiments are considered the “gold standard” of research methods.
 
Experiments are often simple and cheap. Online experiments, for example, can be run by anyone with a website or survey software that allows for randomizing people to control or factor of interest.
 
Of course, experiments that are well-designed provide the most valuable insights. So what goes into designing a good experiment? Check out our article on designing experiments to learn more.