Synthetic Research: The evidence is catching up with the hype

By James Endersby, CEO, Opinium

For the past couple of years, synthetic research has been one of the most talked-about developments in our industry. The promise is seductive: test ideas, explore consumer attitudes and predict behaviour in minutes rather than weeks, at a fraction of the cost of traditional research. For insight teams facing growing demands and tighter budgets, it’s hardly surprising that the technology has attracted so much attention and investment.

And some of that excitement is justified. We’re witnessing extraordinary advances in what artificial intelligence can do. But I think we’re also reaching an important turning point. After a period dominated by impressive demonstrations, ambitious commercial claims and understandable FOMO (fear of missing out), serious evidence is emerging about where these technologies work, where they struggle and where they could lead businesses in the wrong direction.

The conversation is shifting from what synthetic research can produce to what we can actually trust it to tell us. That’s a healthy development for our industry.

Not all synthetic research is created equal

Part of the confusion is that we’re often talking about quite different things as though they’re interchangeable. Synthetic respondents are AI-generated substitutes for people answering research questions. Synthetic personas attempt to represent consumer types, while digital twins seek to simulate individuals or audiences, sometimes using detailed information collected from real people.

Then there are predictive AI models trained on extensive historical human research, and statistical techniques for augmenting datasets. These are different approaches, with different strengths, limitations and levels of supporting evidence. Yet they’re frequently bundled together under the same promise of faster, cheaper insight.

That distinction is important. A system producing a plausible answer isn’t necessarily producing an accurate one, particularly when the research is informing significant business decisions.

The evidence is becoming harder to ignore

One of the most important recent contributions came from the Pew Research Center, which published research in September examining whether AI-generated respondents could replicate the answers of real Americans. The findings should give anyone considering synthetic survey data reason to think carefully.

Across three survey waves, synthetic results differed from human responses by an average of around 12 percentage points, with considerably larger discrepancies on some questions. The systems struggled to reflect the views of certain demographic groups, handled topical issues poorly and produced different results depending on the AI model used.

Perhaps most revealingly, some synthetic respondents appeared more knowledgeable than the people they were supposedly representing. Real people are inconsistent, sometimes poorly informed, influenced by their circumstances and frequently contradictory. That’s not a flaw in human research. It’s part of what we’re trying to understand. If an artificial audience is more rational or predictable than the real one, businesses risk making decisions based on a reassuring but misleading picture of their customers.

Chris Chapman, a respected researcher whose career includes Google, Microsoft and Amazon, has also been challenging assumptions about synthetic survey data. His work raises an important question about validation. Even when a synthetic model successfully replicates previous research, how much confidence does that give us that it will accurately answer a different question, about a different audience, tomorrow?

Meanwhile, the Market Research Society and Royal Statistical Society recently brought together experts from many organisations to examine synthetic data in practice. It’s encouraging to see the conversation moving beyond theoretical possibilities towards actual applications, methodological limitations and the importance of validation.

There are genuine opportunities here

It would be a mistake to treat all this as evidence that synthetic approaches should be dismissed. Some of the most interesting developments are coming from organisations that understand the limitations and are building around them.

Andrew Tindall, for example, have been exploring AI-powered advertising testing grounded in extensive historical research with real people. Their approach makes an important distinction between asking a general-purpose AI model to imagine consumer reactions and developing predictive tools based on observed human responses. Their recent work also acknowledges uncertainty rather than presenting every prediction as equally reliable.

Similarly, work being discussed by Ipsos around synthetic data augmentation and digital twins demonstrates why we need a more nuanced conversation. Using AI to strengthen certain existing datasets, screen early-stage concepts, explore hypotheses or identify areas requiring further investigation is quite different from replacing human respondents and assuming the answers are equivalent.

A better question for insight teams

At Opinium, we’ve always been method-neutral. Our starting point is the client’s problem or opportunity, and then identifying the most appropriate way to address it. Synthetic respondents, personas and digital twins are all part of our toolkit. We’ve experimented with them ourselves and alongside clients, and we’ll continue to do so as the technology evolves.

But we shouldn’t confuse having access to a methodology with having evidence that it’s appropriate for every situation. If you’re exploring early ideas or generating hypotheses, a synthetic approach might be useful. If you’re trying to understand why customers are leaving, whether people will actually buy a product, or how an underserved audience experiences your brand, the risks of relying on artificial responses could be considerably greater.

For client-side researchers considering these tools, I’d suggest asking some straightforward questions. What real human evidence underpins the model? Has it been independently validated against comparable human research? Does that validation apply to my particular audience and decision? How does the system communicate uncertainty? And what happens when it gets things wrong?

A headline claim of 90% accuracy sounds impressive, but it tells us remarkably little unless we understand what was measured and how.

Perhaps we’re missing an even bigger opportunity

There’s another aspect of this debate that deserves more attention. Much of the discussion about AI in research has focused on replacing activities and reducing costs. But what if we approached the opportunity differently?

If AI can make research design, scripting, analysis and reporting faster and more efficient, we have an opportunity to rethink where we invest the time and money saved. We could improve respondent incentives, reach audiences that have historically been difficult to engage, strengthen the representativeness of our samples and spend more time understanding the people behind the numbers.

What if one of AI’s greatest contributions to research is helping us conduct better human research?

I suspect we’re entering a healthier phase for our industry. The early excitement is increasingly being accompanied by independent scrutiny, thoughtful experimentation and a more realistic understanding of what these technologies can and cannot currently achieve. The evidence will continue to evolve, and so should our views. We should embrace innovation, experiment enthusiastically and celebrate genuine advances, while maintaining the standards that make research valuable in the first place.

A synthetic respondent can give you an answer. That doesn’t mean you’ve learned what a customer thinks.

Ultimately, our clients don’t need more answers. They need better evidence on which to make decisions.