RetailOpinion

AI Shopping Agents Fail 43% of the Time. Most Failures Still End in a Purchase.

"AI agents have the potential to power the next era of commerce, but trust is key to making it work."

Answer
AI Shopping Agents Fail 43% of the Time. Most Failures Still End in a Purchase.

The brief

  1. Visa's test of 22 AI shopping models revealed a failure rate of 43%, with models passing only 57.2% of runs.
  2. Despite failures, 54% of the time the AI still completed a purchase, indicating a potential for commerce despite trust issues.
  3. The best-performing AI model passed 82.2% of runs, while the worst passed only 28.9%; models that better uncovered hidden buying factors performed significantly better.

Deployment

Company
Visa
System
Visa Agentic Commerce Benchmark
Stage
Live
pass rate
57.2%

AI agents are being built to shop on people's behalf, but a new test from Visa found they often get it wrong. On November 5, 2026, Visa released the Visa Agentic Commerce Benchmark, which put 22 AI models from 12 labs through 15 realistic shopping challenges.

The brief

  1. Visa's test of 22 AI shopping models revealed a failure rate of 43%, with models passing only 57.2% of runs.
  2. Despite failures, 54% of the time the AI still completed a purchase, indicating a potential for commerce despite trust issues.
  3. The best-performing AI model passed 82.2% of runs, while the worst passed only 28.9%; models that better uncovered hidden buying factors performed significantly better.

Deployment

Company
Visa
System
Visa Agentic Commerce Benchmark
Stage
Live
pass rate
57.2%

The models passed 57.2% of runs. In about 54% of the runs they failed, the AI still made a purchase.

"AI agents have the potential to power the next era of commerce, but trust is key to making it work," said Rubail Birwadker, global head of growth at Visa.

How the Test Works

The benchmark covers categories including TVs, laptops and personal care appliances. To pass a run, a model has to identify a shopper's needs through conversation, uncover every requirement in the task, and make the right decision to buy or not buy. If no product fits, the AI is expected to decline.

In one scenario, a model shopping for a TV had to find out the customer's budget, format compatibility, whether the set supported HDR10+ and its gaming capabilities.

Visa said the requirement to uncover enough information before deciding guards against lucky guesses.

Failures included picking a product that did not meet the customer's needs and buying before finding all of the customer's requirements.

Each model had three attempts at every scenario. All three attempts passed only 34.2% of the time, and models often failed when repeating scenarios they had already passed, according to The Next Web.

Which Models Did Best

Results varied widely. The best-performing model passed 82.2% of runs, and the worst passed 28.9%.

Visa did not name the models in its announcement. According to Fortune's reporting, cited by The Next Web, agents from Google and Meta shared the top score of 82%, a Claude agent came third with 76%, and an OpenAI agent placed sixth with 64%.

Models that were better at uncovering hidden buying factors performed better overall. The top five on that measure passed 70.2% of runs, compared with 44.4% for the bottom five.

Cost also mattered: the five most expensive models per run passed 67.1% of runs, against 50.2% for the five least expensive.

"We're not there yet from a trust or technology perspective," Birwadker told Fortune. "But we'll soon get into a world where consumers can trust and delegate."

The benchmark is publicly available on Visa Payments Frontier, where users can take the test and watch replays of AI attempts.

Visa said it has shared the benchmark with leading model providers, and plans to add more scenarios and product categories and extend testing to checkout, payment and order placement.

United Restricts Agents

Days after Visa's results, United Airlines announced on November 11 that it is restricting access for third-party AI agents, both on its own digital platforms and through online travel agencies and aggregators.

United said the aim is "to prohibit use by unauthorized, third-party personal AI agents or assistants," according to Skift. It is the first US airline to explicitly restrict them.

"I don't want to create another middleman," CEO Scott Kirby said at a Chicago event, as reported by The Wall Street Journal.

United said sharing account credentials with third-party agents can expose sensitive information, and that AI tools could trigger unintended purchases, account changes or MileagePlus redemptions.

It also said automated purchases could conflict with FAA and Department of Transportation rules, pointing to check-in, where a person must confirm their bags contain no prohibited items.

United said it will evaluate the policy as personal AI assistants evolve.

Trust Before Checkout

Not every airline is taking United's approach. Delta CEO Ed Bastian has said he is "cautious" about AI agents in booking, while Southwest has launched a ChatGPT plug-in for flight search.

Consumers are also hesitant. Visa's Trust Index, released in September, found 72% of US consumers had used an AI assistant, but only 23% trusted AI to make payments for them.

Visa's next round of testing will move closer to the point of payment. The company plans to add more shopping scenarios and product categories, and to extend the benchmark to checkout, payment and order placement.

Birwadker said that stage is coming. "We'll soon get into a world where consumers can trust and delegate," he said.

Written by Sachin Mohan

Join the discussion

0 answers

People who run these systems answer here. Your verified role is shown, never your employer unless you choose.

Sign in with LinkedIn to answer

Read next

MachineCon USA · from AIM Media House

Where enterprise AI is decided

An invitation-led day for the CIOs, CDOs and Chief AI Officers who own enterprise AI strategy.

  • New York5th edition · 583 Park Avenue
  • Dallas1st edition

Register for MachineCon USASee MachineCon USA →

  • Keynote on the main stage, MachineCon New York 2025
  • AI150 award winners on stage, MachineCon New York 2025
  • At the tables, MachineCon New York 2025
  • Networking, MachineCon New York 2025