A large-scale study shows why perceived artificiality may be the new creative risk.
Listen to this article:
We’ve all become familiar with AI slop: low-quality content that is so obviously created by AI that we can’t believe it could fool anyone. But more and more, we seem to be confronted with the unsettling possibility that we’re not sure whether what we’re looking at is AI-generated or not. That uncertainty is the starting point for “AI in Disguise—Quasi-Experimental Analysis of a Large-Scale Deployment of AI-Generated Ads,” co-written by HBS AI Institute Frontier Firm affiliate Shunyuan Zhang. She and her co-authors asked what happens when AI-generated visual content enters a real advertising marketplace without being labeled as AI. Do people click less, or more? The result is more nuanced than either AI boosters or skeptics might expect.
Key Insight: A Natural Experiment Inside Real Ad Campaigns
“These ‘sibling ads’ share the same advertiser, landing page, campaign settings, and cater towards the same campaign objective and product.” [1]
The authors’ core inquiry is straightforward: AI can now generate ad images quickly and cheaply, but it has been unclear how those images actually perform. Working with Taboola, a major global advertising platform, they analyzed a dataset spanning more than 369 million real ad impressions and 2.5 million clicks. Taboola released a GenAI Ad Maker tool in July 2023, which allowed advertisers to create images directly inside their platform. Given that advertisers already run informal A/B experiments by running variations of creative assets within a single campaign, the researchers were able to compare ads where the only difference was whether the image was created by AI or by humans.
Key Insight: The Anatomy of Artificiality
“[W]e propose that consumer reactions to ads may vary based on the perceived artificiality of AI-generated content.” [2]
To understand why some AI ads succeed and others fail, the researchers focus on whether people perceived an image to be AI-generated or not. They used over 1,700 images rated by crowdworkers on a five-point scale from “definitely human-made” to “definitely AI-generated,” creating a “looks-like-AI” score for each ad. Consumers associate certain features with AI: highly aesthetic images, vivid color saturation, and lower warmth. But AI-generated ads in the dataset also tend to be clearer and more likely to show large human faces—qualities that consumers tend to interpret as signals of human production. The pattern runs against intuition: the traits that appear regularly in AI-created images are often not the traits people rely on to decide whether something looks like AI. AI, it turns out, is producing content that confounds the very heuristics people use to detect it.
Key Insight: AI Wins When It Escapes Detection
“AI-generated images achieve superhuman CTR levels only when they do not appear to be AI-generated.” [3]
At the aggregate level, AI-generated ads and human-made ads produced statistically indistinguishable click-through rates (CTR). That finding alone is commercially significant: AI content can be created at a fraction of the cost (the platform estimated roughly $0.005 per image) with no measurable performance penalty. When the researchers introduced the “looks-like-AI” score and looked at how that perception combined with the ad’s true origin (AI vs. human), the picture sharpened considerably. Ads that appeared artificial, regardless of whether they actually were AI-generated, saw meaningfully lower click-through rates. Over 45% of AI-generated ads successfully disguised themselves as human-made, achieving higher click-through rates than the group of actual human-made images. Conversely, nearly 25% of human-made ads were mistakenly perceived as AI. Ultimately, AI visuals achieve “superhuman” performance and beat human content, provided they do not look artificial.
Why This Matters
For advertisers, the research suggests treating perceived artificiality as a creative risk and avoiding visual cues that make an ad feel overly machine-made. The same logic applies across organizational functions. In customer service, AI responses may succeed or fail depending on whether customers experience them as natural rather than synthetic. In product design, teams should ask whether AI-generated concepts are realistic and actually useful, not just novel. For business leaders and executives, the takeaway is that AI value can depend on perception. Organizations that can build feedback loops to test how AI outputs are received, not just how cheaply they are produced, will be better positioned to turn AI into a source of strategic advantage.
Bonus
To read more about how AI can imitate humans, and why our ability to detect machine-made content is becoming less reliable, check out Can You Spot the Bot?
References
[1] Exner, Yannick, Jochen Hartmann, Ziqian Ding, Shunyuan Zhang, and Oded Netzer, “AI in Disguise—Quasi-Experimental Analysis of a Large-Scale Deployment of AI-Generated Ads,” Columbia Business School Research Paper No. 5096969 (January, 2025): 9. https://dx.doi.org/10.2139/ssrn.5096969
[2] Exner et al., “AI in Disguise,” 4.
[3] Exner et al., “AI in Disguise, 20.
Meet the Authors
Yannick Exner is a doctoral researcher at the TUM School of Management.
Jochen Hartmann is a professor of digital marketing at the TUM School of Management.
ShunZiqian Ding is a Ph.D. student at the Tepper School of Business at Carnegie Mellon University.
Shunyuan Zhang is Associate Professor of Business Administration at Harvard Business School. She and other HBS faculty contribute to the HBS AI Institute Frontier Firm Initiative.
Oded Netzer is Arthur J. Samberg Professor of Business at Columbia Business School.
Watch a video version of the Insight Article here.