People are gloriously unpredictable. We drive across town to save a little on fuel, then splurge on coffee without a second thought. We swear we'll cancel a subscription the moment the price goes up, and then never do. We tell surveys we're careful with money, and carry card debt month after month.
For marketers, that messiness is the whole job. Every price, every product feature and every line of ad copy is a bet on how real people will behave. Get it right and you grow. Get it wrong and you either lose customers or leave money on the table.
So here's the question many marketing teams are now quietly asking: if AI can write like us, can it predict human behavior too? Can a chatbot tell you how customers will react to a price rise before you make it?
A new study says: partly, and least of all where it matters most.
Take one simple pricing call. Ask a leading AI model how 55-to-64-year-olds in the US would react to a $4 rise in their streaming bill, and it will confidently tell you about half would cancel or downgrade. Ask the people themselves, and 70% shrug and keep paying.
That gap is the story of Human Behavior Bench, a new test from New York startup Rehearsals, built with Google DeepMind and Kaggle. The team surveyed 1,244 Americans, then asked the world's leading AI models to predict how specific groups of them would answer.
What are synthetic respondents?
Traditional market research is slow and expensive. You recruit real people, run surveys or focus groups, wait weeks, and pay for every answer.
Synthetic respondents promise a shortcut. Instead of asking real customers, you ask an AI to answer as if it were them. Tell a model "you are a 58-year-old in Ohio earning $70,000 a year," then ask what it thinks of your new price. Repeat that for hundreds of imaginary people, and you have a survey in minutes, for almost nothing.
The idea has spread fast. Product teams ask chatbots to role-play customer segments. Agencies sell "AI focus groups." Startups build digital twins of shoppers. More and more pricing and messaging decisions are being tested on an AI's idea of a customer instead of a real one.
There's just one problem: hardly anyone has checked whether those imaginary customers behave like real ones. Human Behavior Bench is one of the first large, public attempts to do exactly that.
How the test worked
Rehearsals interviewed and surveyed 1,244 US adults between June and September 2026, collecting more than 95,000 answers. From those, it built 1,199 test questions.
Each question is paired with a specific group of people. Some groups are demographic, like 35-to-44-year-olds earning over $150,000. Others are behavioral, like people who bought a car last year. The AI is told who the group is and asked to predict how their answers will split across the options. That prediction is then compared with what the group actually said.
The team deliberately chose questions where a group answers differently from the national average. So the test isn't whether AI knows what "people in general" think. It's whether AI understands how particular customers differ, which is the thing marketers actually need.
The major models were tested independently on Kaggle, Google's data science platform, which will keep a public leaderboard as new models are released.
The models all scored around 80 out of 100.
Here is how the leading models scored:
Gemini 4 Argon (Google): 80.6
GPT-5.6 Sol (OpenAI): 79.5
Claude Opus 5.5 (Anthropic): 79.3
Gemini 3.8 Flash (Google): 79.1
GPT-6 Astra (OpenAI): 79.0
Grok 4.6 (xAI): 76.2
Rehearsals' own system, Engram 1.0, scored 94.1, though on the company's own testing. More on that below.
Eighty sounds decent. It isn't, once you see where the missing 20 goes.
What 80 actually means
The score is not "right 80% of the time." It measures how closely the AI's predicted pattern of answers matches the real one, where 100 is a perfect match. Because it adds up the error across every option, a model that is off by just a few points on each answer can still lose a lot overall.
A tougher, simpler test tells a starker story. When asked to name a group's single most popular answer, the models got it right about 74% of the time on brand perception and daily routines. On product features, pricing and messaging, that fell to around 41%. Those are exactly the decisions businesses most want help with.
The report lists six real calls a team might make, and every model missed in the same direction:
Can we raise the streaming price by $4? Among 55-to-64-year-olds in the Northeast, AI predicted 36–47% would keep paying. In reality, 70% would.
Should checkout push tap-to-pay? Among shoppers over 60 in the Midwest, AI predicted 3–6% pay by phone or watch. In reality, 25% do.
Should the ad lead with "keeps you dry when you sweat"? Among graduate men earning $50k–$100k, AI predicted 17–23% would find it extremely compelling. In reality, 42% do.
Should the logo go front and center? Among 45-to-59-year-olds earning $50k–$100k, AI predicted 32–48% dislike visible logos. In reality, 14% do.
Are rural drivers renewing on autopilot? AI predicted 50–56% renew car insurance without shopping around. In reality, 24% do.
Should we market a balance-transfer card? Among graduate men in the West, AI predicted 13–26% see carrying card debt as normal. In reality, 37% do.
Trust the AI on any of these and you would make the wrong call: price too cautiously, ignore a payment method a quarter of older shoppers already use, or bury a message that lands with four in ten buyers.
Wrong in the same direction
The misses aren't random. The models consistently imagine people as calmer, less committed and more indifferent than they really are. They underplay the strongest feelings by about 13 points, and they overpredict "never" and "don't" answers.
A few examples from the report:
One in four shoppers over 60 in the Midwest already pays by tapping a phone or watch. The AI guessed 6% at most.
42% of men earning $50k to $100k found "keeps you dry when you sweat" an extremely compelling pitch for workout clothes. The AI said under a quarter.
84% of high-earning women under 30 watch mostly on their phones. The AI guessed about half.
The age blind spot
Age is where the distortion is sharpest. The models usually know which way an age group leans, but not by how much. For people under 60, they capture only about three-quarters of how different the group really is. For people 60 and over, they overshoot, and that is the group they predict worst.
The errors read like stereotypes. The models guessed more than half of people over 65 had skipped TikTok, Reels and Shorts in the past week. The real figure was one in five. They guessed up to 42% of older Midwesterners find shows through autoplay. Only 9% do. In other words, AI imagines young people as a little more ordinary than they are, and older people as a lot more old-fashioned.
Asking a second AI engine won't help The six models agree with each other about twice as closely as they agree with real people. Getting the same answer from ChatGPT, Gemini and Claude isn't confirmation. It may just be the same blind spot, three times over.
The catch
Rehearsals sells a customer-simulation product, and its own system tops the table at 94.1. That score came from the company's own testing rather than Kaggle's, so treat it as a claim for now. The data is also what people said, not what they bought, and it covers only US adults.
Still, the core finding doesn't depend on Rehearsals' product. Six models from four different companies, tested independently, all missed in the same direction. That is hard to put down to a rigged test.
The authors flag other limits too. The benchmark predicts groups, not individuals. The typical question rests on about 176 people, so small gaps between models may just be sampling noise. And outside companies can't yet submit their own systems, though Rehearsals says that is coming. A larger round with 3,500 respondents is due in the next few weeks, followed by a version covering ten countries, including the UK, Japan, Korea and Singapore.
Why it matters
These models are trained to be polite, balanced and reasonable. Customers are none of those things. The insight that changes a strategy usually hides in the lopsided, emotional answer nobody expected, which is exactly what AI smooths away.
Use AI to think faster. But before you bet the budget, ask the people.
Sources: Human Behavior Bench, Rehearsals · Leaderboard, Kaggle
