Overview
Google is arguably the most well-known search engine in the world. It holds over 90% of the search engine market share worldwide. If you browse the internet, there’s a good chance you use Google.
Like most other tech companies that don’t charge end-users for their core product, Google relies on advertising revenue to make money. And make money it has. Google’s advertising business generated almost $150 billion in revenue in 2020.
We know ads are being bought. For example, it’s very common to see Google search ads with links to advertised websites appearing before organic search results. But are they effective? The sheer volume and history of such ad purchases would suggest they must be at least somewhat effective. But go onto social media and you’ll find that posts with ads get significantly fewer likes and shares than other posts.
In this spirit, we designed a simple experiment to test whether users would be more or less likely to click on a link to a website from Google’s search results if it was labelled as an ad vs. an organic search result.
The Experiment
We recruited 400 people from Amazon Mechanical Turk to view a preview of a set of Google search results with the goal of selecting a website to hypothetically buy a t-shirt. We randomized whether the first link was labelled as an ad or not, then asked participants which link they’d click. Both links are real links from real t-shirt companies.
Participants were instructed to “Imagine you’re Google searching for t-shirts to buy, and you get the following search results (pictured below):” Participants then viewed the search results, except we randomized whether they saw the “Ad” label and indented URL with the first link (i.e., Zazzle).
Google is arguably the most well-known search engine in the world. It holds over 90% of the search engine market share worldwide. If you browse the internet, there’s a good chance you use Google.
Like most other tech companies that don’t charge end-users for their core product, Google relies on advertising revenue to make money. And make money it has. Google’s advertising business generated almost $150 billion in revenue in 2020.
We know ads are being bought. For example, it’s very common to see Google search ads with links to advertised websites appearing before organic search results. But are they effective? The sheer volume and history of such ad purchases would suggest they must be at least somewhat effective. But go onto social media and you’ll find that posts with ads get significantly fewer likes and shares than other posts.
In this spirit, we designed a simple experiment to test whether users would be more or less likely to click on a link to a website from Google’s search results if it was labelled as an ad vs. an organic search result.
The Experiment
We recruited 400 people from Amazon Mechanical Turk to view a preview of a set of Google search results with the goal of selecting a website to hypothetically buy a t-shirt. We randomized whether the first link was labelled as an ad or not, then asked participants which link they’d click. Both links are real links from real t-shirt companies.
Participants were instructed to “Imagine you’re Google searching for t-shirts to buy, and you get the following search results (pictured below):” Participants then viewed the search results, except we randomized whether they saw the “Ad” label and indented URL with the first link (i.e., Zazzle).
|
|
Participants were then asked, “Which link would you click?” with answer options, “Zazzle [(ad)],” “TeePublic,” “None, close the browser.”
Results
Analyzing the results using a chi-squared test of independence, we found that viewers were significantly less likely to select the Zazzle t-shirt link when it was an ad (31.5% clicked) versus an organic search result (47.8% clicked), (p = 0.001). This 16.3 percentage point reduction equates to a 34% drop in clicks. We found no difference for rates of closing the browser (9.5% vs. 10.9%, p = 0.633).
Conclusion
If you operate a website, the results suggest a significant advantage to organic search traffic relative to ads. Admittedly, large volumes of organic search traffic are much harder to attain, which might be why ads are still in high demand. Nevertheless, the moral of the story remains the same: organic search beats ads.
Methods Note
We used a chi-squared test of independence to test for significant differences in selecting the first search result between our ad and organic conditions. For significant differences, the difference between the two groups' rates would be large and its corresponding “p-value” would be small. If the p-value is less than 0.05, we consider the difference statistically significant, meaning we'd likely find a similar effect if we ran the study again with this population.
Methods Note
We used a chi-squared test of independence to test for significant differences in selecting the first search result between our ad and organic conditions. For significant differences, the difference between the two groups' rates would be large and its corresponding “p-value” would be small. If the p-value is less than 0.05, we consider the difference statistically significant, meaning we'd likely find a similar effect if we ran the study again with this population.
Popular Experiments
COVID MasksDoes wearing a COVID mask affect how others think of you?
|
Video GamesAre video games more enjoyable than board games?
Does age or gender matter? |
Zero-Sum PoliticsDo Democrats or Republicans view society as win-lose?
|