How AI shopping agents know your reviews are real

When a shopper asks an AI agent for "a good dog harness," the agent has seconds to choose a few products out of hundreds. Price, availability and the listing's own words get it to a shortlist. Reviews are often what tips it from there: a 4.8 from 16,000 reviews says something a product description can't.

That only works if the reviews are real. So it's fair to ask how an agent knows they are. The honest answer is that the agent mostly doesn't check for itself. It trusts the catalog it searched, and the catalog trusts whatever verified the review before it got there.

Why reviews matter to agents

Reviews are one of the few signals in a listing that the merchant didn't write. An agent can't hold your product or try it on, so the experience of people who bought it stands in for that.

OpenAI says so directly. Its help page on shopping in ChatGPT lists reviews among the things ChatGPT may consider when choosing products, and says it can show review summaries drawn from public websites. The same page also says plainly: "Reviews and ratings are not verified by OpenAI."

That's the pattern. The agent uses reviews, but verification happens upstream, before the data reaches it.

Where an agent's review data comes from

For Shopify stores, agents search public shopping catalogs through their MCP servers. Every product they return carries a rating like this one, from a search for "dog harness":

"rating": { "value": 4.8, "scale_min": 1, "scale_max": 5, "count": 16030 }

That's a value, its scale and a count. Products and variants are rated separately; the same harness's featured variant showed 4.8 from 938 reviews. The catalog's docs tell agents to "keep each score with its scale and review count, and say which one you're showing," and let them filter for things like "a rating of 4 or more from 20 or more reviews."

What the catalog doesn't return, at least in what we've seen, is the review text, a per-review "verified" flag, review dates or the spread of stars. An agent working from search results sees an average and a count, nothing more.

Those numbers come from Shop's product reviews. According to Shopify's help center, "Only Shop customers that have purchased a product from you are allowed to submit a review," and buyers are asked for one after their order is delivered. Brands that use a review app can also feed those reviews in: Shopify lists Judge.me, Loox, Okendo, Stamped.io and Yotpo as supported partners, and their reviews count toward Shop's average ratings once synced.

How real reviews are told from fake

They're tied to an order. A review from a partner app only shows on Shop if it's "linked in metaobjects to a verified customer with a Customer ID, Order ID, and Product ID." Without all three, it's left out. Reviews imported from a CSV, including ones moved between review apps that way, aren't eligible either. Reviews collected in Shop carry a "Verified by Shop" badge on your store that you can't remove.

They're screened against rules. Shop checks reviews before publishing them: one review per purchase, no one with a conflict of interest (you, your staff, friends, family or competitors), nothing "artificially created by a bot or other automation," no ads or links. Merchants can't pay for reviews in money, discounts, free products or refunds, can't ask only happy customers, and can't pay someone to change or remove a review. Shop's own requirements can be stricter than your store's, so Shop may show fewer reviews than your site does.

Faking them is illegal in the US. The FTC's rule on consumer reviews and testimonials took effect on October 21, 2024. It bans fake reviews (including AI-generated ones), buying reviews, paying for reviews that say something positive or negative, undisclosed insider reviews, company-run "independent" review sites, suppressing negative reviews through threats or by hiding them while implying you show them all, and buying fake followers or views. The rule as published puts the current maximum civil penalty at $51,744 per violation, though courts decide the actual amount.

Volume matters, at least in our own ranking. A 5.0 from three reviews isn't the same as a 4.7 from 3,000, and the catalog's own filters let agents set a minimum count. On Bark Twice's lists, like the best shoes, we pull every rating toward a typical 4.2 until it has about 20 reviews behind it, so one five-star review can't beat a product with hundreds. That's our formula, and we publish it so you know. It isn't the catalog's.

What isn't public

Nobody outside these companies can tell you exactly how much a given agent weighs reviews against price, relevance or anything else. Shop doesn't publish its ranking formula. OpenAI says reviews are among the things ChatGPT considers, without saying how much. We couldn't find public documentation of how Gemini, Claude or Perplexity weigh review counts, recency or verified-purchase status when recommending products, so we won't guess.

What we can say: whatever an agent does with reviews, the ones reaching it through this catalog have already passed the purchase link and screening above.

What to do about it

1. Collect reviews through Shop or a supported app. Judge.me, Loox, Okendo, Stamped.io or Yotpo, synced to Shop with "Display reviews from partner apps in the Shop app" turned on. Reviews that never reach the catalog don't help you there.
2. Ask every buyer, the same way. A neutral request after delivery, sent to everyone. Don't filter for happy customers, and don't offer a discount for a review.
3. Never buy, write or gate reviews. It breaks Shop's rules and the FTC's, and Shop's checks are built to leave those reviews out.
4. Don't move reviews around by CSV. Imported reviews aren't eligible on Shop. If you switch apps, follow the partners' migration guidance.
5. Reply to the bad ones and fix what they say. Shop's guidelines cover how merchants reply to reviews. A complaint that repeats is usually a listing or product problem you can fix, and a fair reply tells the next shopper you're paying attention.
6. Keep reviews on one product, not split across copies. Selling each color as its own product spreads reviews thin, and each copy looks less proven than one product would.
7. Mark up reviews on your own pages too. Standard review structured data on your product pages helps any agent or search engine that reads your site directly, not just the catalog.

What Bark Twice shows

Our report shows your products the way agents get them from the catalog: the rating and review count on each product and its featured variant, where low counts are holding a product back, and which of your products show up with no rating at all. It costs $99, then $10 a month to keep tracking it as your reviews change.

Want to know where your own store lands? Check it free, or get a full report.