# Synthetic Customers for DTC Brands: What the Colgate AI Shopper Paper Means for Your Meta Ads, Pricing and Product Tests

Colgate-Palmolive and PyMC Labs showed that LLM personas can reproduce purchase intent surveys at about 90% of human test-retest reliability, if you ask them to write instead of rate. What that unlocks for a DTC brand: screening ad angles before Meta spends on them, ranking bundles and price points before a shoot, pulling objections for the landing page, and where the method breaks. With the exact prompt-and-score recipe and what it costs.

Author: Nikhil Sharma (https://nikhil.pro/page/about/)
Published: 2026-11-21
Updated: 2026-11-21
Canonical: https://nikhil.pro/synthetic-consumers-llm-purchase-intent/
License: CC BY 4.0, attribution required.

Every DTC owner has the same three questions and pays for the answers in the slowest way possible. Which of these bundles should we shoot? Which ad angle do we put budget behind? Is $59 too much? The usual answer is to launch all of it and let Meta and the customers tell you, which costs a month and a few thousand dollars per question.

In October 2025 Colgate-Palmolive and PyMC Labs published a paper that makes a cheaper first pass credible. They built panels of LLM personas, matched each one to a real survey respondent, and checked the synthetic answers against 9,300 real ones across 57 product concepts. With the right method, the synthetic panel tracked the humans at about 90% of the human panel's own test-retest reliability. With the obvious method, it was useless.

This post is about what that means for a brand running Shopify and Meta, what you can do with it this week, and where it will lie to you.

**Short version:** build a panel from your own customer mix, ask it to write reactions instead of giving ratings, score the writing, and read the objections. It will rank your concepts, angles and price points well enough to decide what to test for real. It will not tell you CPA, elasticity, or anything about a market the model has not seen.

## What the paper found, in one minute

The data was 57 concept surveys for personal care products, 150 to 400 US consumers each, asked purchase intent on a 1 to 5 scale. The researchers gave GPT-4o and Gemini 2.0 Flash the same concept and the same demographic profile as each respondent, and collected the answer three ways.

| How the model was asked (GPT-4o) | Share of human test-retest ceiling | Distribution match (0 to 1) |
|---|---|---|
| Rate it 1 to 5 | 81.7% | 0.26 |
| Write a reaction, second call turns it into 1 to 5 | 84.7% | 0.72 |
| Write a reaction, score it by semantic similarity (SSR) | 90.2% | 0.88 |

The first row is what most "AI focus group" tools do. The ranking of concepts was roughly right, but the answers were almost all 3s, with no 1s or 5s. The third row is the paper's method: the model writes a few sentences, the text is embedded and compared with five reference sentences from "I would definitely not buy this" to "I would definitely buy this", and the similarities become a distribution over the scale.

[Image: Semantic Similarity Rating in four steps: a persona and a product concept go to the model, the model writes a short reaction, the text is embedded and compared with five reference sentences, and the similarities become a distribution over the 1 to 5 scale]

> The finding is about how you ask, not which model you use. Same persona, same concept, same model. The only difference between a useless 0.26 and a usable 0.88 is writing first and scoring after. If a tool you are paying for asks the model for a number, that is the number it is giving you.

Three more results that matter for how you use it:

- **Demographics did the work.** Without persona details the correlation fell to about half the ceiling. The panel has to look like your customers.
- **Age and income curves reproduced; gender and region did not.** Younger and older respondents were less likely to buy, both for real people and for personas. Gender and region effects were small in the human data and reproduced less well.
- **The panel beat a trained model.** A gradient boosted model trained on the demographics and concept features reached 65% of the ceiling. The LLM panel reached 88% with no training on the survey data at all.

## What this unlocks for a DTC brand

### Meta ads: screen angles before Andromeda does it for you

The expensive part of creative testing is not production. It is the week of spend while Meta works out which three of your fifteen ads deserve delivery, and the [creative similarity](https://nikhil.pro/meta-ads-creative-similarity-explained) problem where ten of the fifteen were the same angle in different clothes, so they competed with each other instead of expanding reach.

A synthetic panel is good at the step before that. Write the hook and the first two lines for each angle: gifting, self treat, occasion, ingredient story, price anchor, social proof. Show each persona the angle, not the finished creative, and ask for a written reaction. Then:

- **Rank the angles** by scored intent across the whole panel. You will usually find two that are clearly ahead and two that nobody reacts to.
- **Read the objections per angle.** The "self treat" angle gets "I would not spend this on myself" from the over 45 personas and "would buy for my mum" from the under 30s, which tells you the angle works, but aimed at the wrong buyer.
- **Check that the winners are different from each other.** If your top three angles produce the same reactions from the same personas, they are one angle and Andromeda will collapse them into one entity. Pick winners that split the panel differently.

What it cannot do: predict CTR, CPC or CPA. The persona never sees the video, the thumbnail, the feed it sits in, or the auction. It reacts to the idea. Put budget behind the top three ideas and let the real test pick the execution.

### Bundles and new products: rank six before you shoot two

Holiday is the usual case. Six bundle ideas, budget to photograph two, a founder with a favourite. A 300 persona panel built from your customer age and location mix ranks the six in an afternoon. The value is rarely the top two, which the team already agrees on. It is the bundle that lands fifth with the same objection repeated across segments, which you quietly drop before anyone spends on it, and the one that lands third but with no objections at all, which is the safer second shoot.

### Pricing: order the steps, then test two

The paper is clear that synthetic consumers do not capture budget constraints, so do not ask for elasticity. Ask for ordering. Show the same box at $49, $59 and $69 and look at where written intent drops hardest and in which segment. If the $59 to $69 step is where the under 35 personas fall off and the over 45s do not move, you have a hypothesis: test $59 against $69 on a Meta campaign aimed at the older segment first, where the risk is lowest. Interesting detail from the paper: both humans and personas rated the higher price tiers above the lowest, so a concept survey reads price partly as a quality signal. A real checkout does not.

### Landing pages: get the objections before the bounce rate tells you

The number is the least useful output. The written reactions, grouped by segment, give you the three or four reasons someone would not buy, in words a shopper might use. For a brand that ships gift boxes, expect "who is this for, exactly", "will it arrive fresh" and "that is a lot for something I cannot taste first". Each one maps to a page element: a "perfect for" block, a shipping and freshness callout above the fold, a smaller trial size. You get the list before the first paid click, instead of reading it off a Hotjar recording three weeks later.

### Email and offers: pick the subject line and the offer shape

The same panel reacts to offer framing. Free shipping over $75 versus 10% off versus a free add on. The personas will not tell you open rate, but they will tell you which offer the gifting segment ignores and which one the repeat buyer segment reads as a discount on something they were going to buy anyway. That narrows a Klaviyo A/B test from five variants to two.

### Agencies and freelancers: a deliverable that costs nothing to produce

If you run ads for clients, this is a one afternoon artefact that most clients have never seen: here are your six angles ranked by a panel built from your own customer file, here are the objections each one raises, here are the three we are putting budget behind and why. It frames the creative test as a decision already made for reasons, instead of a lottery.

> Present it as a hypothesis list, never as a forecast. "The panel scored this bundle 3.8" lands in a meeting as a prediction and it is not one. "The panel ranked these six in this order and raised these objections, so we are testing the top two" is what the paper supports.

## Where it will lie to you

**Your market, if it is not the national average.** The model has strong priors about a national US gift market and weak ones about a regional one, a niche one, or an international one. It answers from the strong priors. Local shipping expectations, local occasions, local price anchors come back generic. The authors say directly that the method "will not conjure valid consumer preferences" where background knowledge is thin, and a small brand hits that on day one.

**Anything the persona cannot see.** Creative execution, the feed, the auction, your competitors' ads that week, the customer's bank balance in the last week of the month. Every one of those moves a real result and none of them exists for a persona.

**Segments the paper itself could not reproduce.** Gender, region and ethnicity effects were weak in the synthetic data. If your product's whole thesis is that one of those segments behaves differently, the panel is the wrong tool.

**The reference sentences.** SSR scores against five hand written anchor sentences, and the paper's were tuned on its own 57 surveys. Different anchors give different scores. Write several sets, average them, and never compare scores from two runs with different anchors.

**Positivity.** The paper reports that synthetic reactions were longer, more specific and less positively biased than the human open ended answers, on the basis of quoted examples. Treat that as encouraging, not proven. A persona that writes three polished sentences about fragrance longevity is not evidence a shopper cares about fragrance longevity.

## How to run it

The version worth running, in order of effort.

**1. Build the panel from your customers, not a census.** Pull age band, state and order value band for your last 2,000 customers from Shopify. Sample 300 personas in those proportions. The paper's result collapsed without demographics, and your customer mix is the demographic that matters. If you only have location, start with location and the age mix from your Meta audience insights.

**2. Ask for writing, never a number.** The prompt is short: here is who you are, here is the product and price (or the ad angle, or the offer), write three or four sentences on whether you would buy and why. Temperature around 0.5. Two samples per persona, averaged, as in the paper. Include the concept image if you have one; text only was slightly worse.

**3. Score the writing afterwards.** Two options. The cheap one is the paper's follow up method: a second call reads the paragraph and assigns 1 to 5, which reached 85% of the ceiling. The better one is SSR: embed each paragraph, embed five reference sentences from "I would definitely not buy this" to "I would definitely buy this", take cosine similarities, subtract the minimum, normalise. Write three or four anchor sets in different wording and average them.

**4. Validate once against something real.** Before trusting the panel on a new concept, run it on products you already sell and compare its ranking to your actual sales mix and review scores. If it cannot order the products you have, it will not order the ones you do not.

**5. Keep the text.** Group reactions by segment and read them. That is where the page copy, the angle fixes and the test ideas come from. The score only tells you which group to read first.

| Step | Cost for 300 personas, one concept |
|---|---|
| 600 short completions, mid tier model | roughly $2 to $4 |
| 600 embeddings plus anchor sentences | under $0.10 |
| Building personas and anchors from your data | an afternoon, once |

## Verdict

The LinkedIn version of this paper says a toothpaste company killed market research. The paper says something narrower and more useful: if you ask LLM personas to write instead of rate, they track real consumers closely enough to be worth running as a first screen, in a category the model knows, for a company with human data to check against.

For a DTC brand that translates into a cheap habit. Before a shoot, a creative test, a price change or a landing page rebuild, run the panel, read the objections, and spend the real budget on whatever it could not settle. Nobody at Colgate stopped running surveys. Nobody should stop running Meta tests either. They should just start them with better candidates.

> How this was written. From the paper itself (arXiv 2510.08338, version 3, October 2025), including its appendix and limitations section, and from working the prompting and scoring steps against current model APIs. The figures in the tables are the paper's. The cost estimates are mine at current list prices and will drift.

Related reading: [why Meta only spends on a few of your ads](https://nikhil.pro/meta-ads-creative-similarity-explained) for what happens to similar angles once they reach the auction, [six AI workflows I run on a Shopify store, and what broke](https://nikhil.pro/ai-workflows-for-shopify), and [the most useful AI I built was not customer facing](https://nikhil.pro/most-useful-ai-not-customer-facing).

## Frequently asked questions

**What is a synthetic customer panel?**

A set of LLM personas, each given a demographic profile that matches one of your real customers, shown a product, price or ad concept and asked to react. The Colgate-Palmolive and PyMC Labs paper tested panels like this against 9,300 real survey responses across 57 personal care concepts and found they tracked the human results at about 90% of the human panel's own test-retest reliability, when the reaction was collected as written text and scored afterwards.

**Can I use AI personas to test Meta ad angles before launching?**

As a screen, yes. Show each persona the hook and the first two lines of body copy for each angle and ask for a written reaction, then score and group the reactions. It ranks angles and surfaces the objection each one triggers. It does not predict CTR or CPA, because it cannot see the creative execution, the auction or the feed around the ad. Use it to decide which three angles to put real budget behind, not to skip the test.

**What is Semantic Similarity Rating (SSR)?**

The scoring method from the paper. Instead of asking the model for a number, you ask for a short written reaction, embed the text, and compare it by cosine similarity with five reference sentences, one per point on a 1 to 5 scale. The similarities become a probability distribution over the five points. Averaging several sets of reference sentences makes the result less sensitive to wording. Asking the model for a number directly produced mostly 3s and a distribution match of 0.26; SSR reached 0.88.

**Can a synthetic panel replace a real survey or a Meta test?**

No. The paper's results come from one company's 57 personal care surveys for the US market, with reference sentences hand tuned on that set. The authors say the method depends on the model already knowing the category, that gender, region and ethnicity effects reproduced poorly, and that it cannot capture budget constraints or marketing exposure. It is a cheap first filter that makes the real test smaller and better aimed.

**Can it tell me the right price for my product?**

It can order price points and tell you which step produces the sharpest drop in written intent, and from which customer segment. It cannot give you elasticity or a revenue forecast; the authors list budget constraints as something synthetic consumers do not capture. Use it to pick which two prices to test for real.

**How much does it cost to run?**

A 300 persona panel at the paper's settings is 600 short completions and 600 embeddings, which is a few dollars per concept at current list prices for a mid tier model. The real cost is an afternoon building personas from your customer data and writing reference sentences for your category.

**Which models did the paper use?**

GPT-4o and Gemini 2.0 Flash for the written reactions, at temperature 0.5, two samples per persona averaged, and OpenAI's text-embedding-3-small for scoring. No fine tuning and no training on the survey data.

**Where does the method fail for a small brand?**

Anywhere the model has thin background knowledge: a new category, a regional market, a brand with no public footprint, or a customer base that does not look like the national average. The personas answer from the nearest thing the model knows, confidently. Validate against products you already sell before trusting it on products you do not.


## Work with the author

- Thirty minutes on a call: Which angles, bundles or prices you are weighing, what customer data you have, and whether a synthetic screen is worth running before the real test. https://nikhil.pro/meeting
- Which task to automate first: A one page scoring method for deciding where AI earns its keep in a Shopify operation. https://nikhil.pro/which-task-to-automate-first
- All services: https://nikhil.pro/page/about/