Visa Tested 22 AI Shopping Agents. The Payment Stack Wasn't Part of the Test.

You've seen Visa's headline number: 57.2% pass rate across 22 AI shopping models. Almost half of all test runs failed. The coverage reads as a report card for AI shopping agents, with scores, rankings, and the predictable takeaway that agents aren't ready yet.

But the benchmark tested whether agents understood what to buy. It did not test whether they could pay for it, settle the transaction, or handle the fallout when they got it wrong. The 42.8% of runs that failed didn't just fail quietly. In more than half of those failures, the agent went ahead and bought something anyway.

The payment stack downstream of that buy button has no way to distinguish a correct agent purchase from an incorrect one. Correct and incorrect purchases alike arrive as properly authorized card transactions, settle through the same clearing process, and land on the same merchant ledger.

The difference between a 57.2% pass rate and a 100% transaction completion rate is a gap the existing payment infrastructure can't see, and a cost somebody has to absorb.

What Visa actually measured

Visa Research built 15 shopping episodes, each structured as a conversation between two AI models. One plays the customer, holding a private dossier of requirements, some stated upfront and some hidden until the shopping assistant asks the right questions. The other plays the shopping assistant, the model being tested.

The assistant has to investigate what the customer actually needs, search a simulated market that's identical for every model, and recommend the right product. A run passes only when the product meets every requirement in the answer key and the assistant uncovered each requirement during the conversation. Getting lucky doesn't count.

22 models from 12 labs ran each episode three times. That's 990 total runs.

The top performers, Gemini 4 Argon and Muse Wonder, each hit 82.2%. The bottom, Thinking Machines' Inkling, landed at 28.9%. Costlier models did better on average: the five most expensive per run passed 67% of the time, while the five cheapest passed 50%.

But the confidence intervals tell a different story than the leaderboard. The 95% CI for the top two models runs from 69% to 91%. The next eleven models' confidence intervals all overlap that range. After 990 runs, there is no statistically clear leader.

The 23% that bought anyway

The overall results break into three buckets: 57.2% passed, 23.1% ended in a wrong purchase, and 19.7% ended in some other outcome, usually the customer declining because the assistant said nothing fit, or the run hitting its tool-call limit before reaching a decision.

That 23.1% is the number that matters for payment infrastructure. Each of those 229 runs produced a transaction that looks perfectly normal to the acquiring bank, the card network, and the issuer, because the product was ordered, the payment was authorized, and the merchant fulfilled it. Nothing in the transaction data signals that the agent recommended a TV without HDR10+ support when the customer's kid needs it for Xbox clips.

Think of it like a personal shopper at a department store who walks up to the register with an armful of clothes in the wrong size. The cashier doesn't know the sizes are wrong, the receipt doesn't know, and the credit card statement doesn't know. The only person who knows is the customer, and they find out when they get home and open the bag.

At scale, each of those wrong purchases is a return, a dispute, or a chargeback waiting to happen. The merchant delivered what was ordered, but the customer didn't get what they needed. And the chargeback system has no reason code for "the AI misunderstood my intent."

Failure mode matters more than pass rate

A single accuracy number hides the most expensive variable in the benchmark data. How a model fails determines how much it costs the payment system.

Fable 5.1 passed 75.6% of its runs. When it failed, it bought every single time. Eleven failures, eleven wrong purchases, zero cases where it held back and said nothing fit. If you're a merchant or an issuer, every Fable failure is a transaction you'll hear about later.

Muse Wonder passed 82.2% of its runs. When it failed, it mostly didn't buy. Of its 8 failures, only 2 were wrong purchases. The other 6 were cases where the customer declined or the run stalled, outcomes where no money changed hands.

From a payment infrastructure perspective, Wonder at 82.2% with 2 wrong purchases is dramatically cheaper than Fable at 75.6% with 11. The 6.6 percentage points of accuracy gap between them understates the real difference, because the downstream cost of a wrong purchase dwarfs the cost of a non-purchase.

GPT-6.1 Sol shows the same pattern from a different angle. It passed 62.2% of runs but produced only 2 wrong purchases out of 17 failures. The remaining 15 failures were non-purchases: the model either stalled or talked the customer out of buying. Sol's accuracy is 20 points below the leaders, while its wrong-purchase rate is comparable to Wonder's.

Issuers building fraud models and merchants calculating dispute reserves need to know which model doesn't buy when it's wrong, and no model provider publishes that metric. Visa's benchmark is the first dataset where you can derive it.

The consistency problem

A model passed all three runs of a given task only 34.2% of the time.

That number means the same shopping request, sent to the same model with the same market, produces different outcomes on different days. Not different product recommendations within the correct set, but different pass/fail results. A model that passed Episode 7 on Monday might buy the wrong thing on Tuesday and stall on Wednesday.

For the payment stack, inconsistency is a distinct problem from inaccuracy. Fraud detection systems build behavioral profiles from spending history. If a cardholder's agent buys within predictable categories and amounts, velocity checks and anomaly detection can establish a baseline. But when the same agent makes substantively different purchasing decisions on repeated attempts, the spending pattern itself becomes noisy.

Fraud models tuned for human spending patterns assume a degree of behavioral stability. You buy coffee in the morning, groceries on weekends, and the occasional electronics purchase lines up with your income. An agent that passes a task two out of three times introduces a kind of noise that isn't fraud and isn't consistent behavior. It's a third category the models weren't designed for.

Where the benchmark stops

The benchmark tests discovery and intent. Can the model figure out what the customer needs and recommend the right product? That's Phase 1.

The buy button is the boundary. Everything that happens after it, the entire machinery that turns a purchase decision into settled funds in a merchant's account, sits outside the test.

Once the agent presents a credential, the acquirer routes it through the card network and the issuer approves or declines. The benchmark doesn't test whether agents can handle 3D Secure challenges, present tokenized credentials, or navigate the authentication flows that Visa and Mastercard have built differently for agent payments.

After authorization, the transaction enters clearing. Batch files move between acquirer and issuer, and funds settle on a timeline measured in business days. An agent that reasons at inference speed has to interact with infrastructure that moves at settlement speed.

And when the customer realizes the agent bought the wrong thing, somebody has to unwind the transaction. The evidence architecture for agent disputes barely exists. The spending guardrails that could catch wrong purchases before they settle are still being built.

Visa has announced Phase 2 will cover checkout and payment. Phase 3 will address post-checkout. Neither has shipped.

What Frontier Commerce already revealed

Visa didn't just build a benchmark. In May 2026, Visa and Andon Labs gave an AI agent named Dee a physical store, three AI supplier agents, and a set of real Visa cards. Dee was told to run the store. It was not told how.

Dee chose the 2% rewards credit card for 96% of its procurement payments. It negotiated with supplier agents over email, pushed back on pricing, and managed its own credit balances. On the surface, it looked like an agent that understood money.

The cracks showed on day one. Dee tried to repay its credit balance immediately after its first customer sales. The payment was blocked because batched settlement hadn't cleared the customer's payment into Dee's account yet. The agent reasoned about money at the speed of an API call, but the money itself moved at the speed of batch settlement, hours or days later.

That gap, agent-speed reasoning against settlement-speed money movement, doesn't show up in a benchmark that stops at the buy button. It only shows up when real money is on the line.

Visa's red team uncovered a split in Dee's defenses. Dee was susceptible to social engineering through its customer chat, falling for a "security test" ploy and a bulk-pricing exploit. Yet the payment layer underneath held: double-payment deduplication caught duplicate charges, fabricated supplier IDs were blocked at the payment rail, and a supplier agent independently refused to ship when the payment attribution didn't match the order.

The benchmark measures whether agents understand intent, but Frontier Commerce reveals what happens when agents that understand intent collide with infrastructure that cares only about authorization and settlement.

What Phase 2 will need to solve

When Visa extends the benchmark to checkout and payment, the test surface expands from a single question (did the agent pick the right product?) into a chain of dependent steps, each with its own failure modes.

An agent needs to submit payment without storing raw card numbers, which means tokenized credentials, scoped authorization, or a protocol like Mastercard's Agentic Tokens that bakes spending constraints into the token itself. The benchmark will need to test whether agents can navigate these flows without human intervention.

3D Secure adds another layer. It was built for a human sitting at a screen, entering a one-time code from their phone, and an autonomous agent can't complete that ceremony. Visa's Payment Passkeys offer one path, authenticating the cardholder via FIDO2 biometrics when the agent is set up, then letting the agent transact on that prior authentication. Mastercard authenticates at the mandate level instead.

Phase 2 will have to test both approaches, or define which one the benchmark assumes.

Then there's what happens after the purchase fails. The current benchmark produces a clean outcome: pass, wrong purchase, or no decision. A Phase 2 benchmark needs to define what happens when the wrong purchase enters the payment system.

Does the test include a return flow, a dispute flow? How does the benchmark score an agent that buys the wrong thing but self-corrects by initiating a return before the customer notices?

The 23.1% wrong-purchase rate from Phase 1 becomes the input to Phase 2. Those 229 runs are the test cases for whether the post-transaction infrastructure can handle agent-initiated errors at scale.

The cost nobody is measuring

Visa built the first rigorous, publicly available benchmark for agent shopping. The methodology is sound: controlled markets, hidden requirements, triple runs for consistency, confidence intervals that don't pretend a leaderboard is meaningful when it isn't. The model-by-model data is the most granular public dataset on agent shopping performance that exists.

What the benchmark doesn't produce is a cost model. A wrong purchase from Fable at $1.82 per run isn't just a test failure. In production, it's a chargeback (industry estimates put merchant-side dispute costs in the tens of dollars per incident before considering return shipping, restocking labor, and the customer who may never let the agent shop again). A non-purchase from Sol at $0.38 per run is a lost sale, nothing more.

The industry needs a second metric alongside pass rate, one that measures how often a model fails in a way that costs money downstream. Call it wrong-purchase rate. Visa's data makes it derivable for the first time, and it should be a headline number, not something you have to calculate from the raw tables.

Until the benchmark reaches checkout and settlement, every pass rate is provisional. A model that picks the right product 82% of the time but can't navigate 3D Secure or present a tokenized credential has a production pass rate of zero. Visa built a rigorous test for whether agents understand what you want, but nobody has built the equivalent test for whether they can actually pay for it.