Visa tested AI shopping agents – here’s how they performed
The new Visa Agentic Commerce Benchmark indicates leading artificial intelligence models have room for improvement as shopping tools.
Under the Visa benchmark, AI models are required to identify the needs of a consumer based on a conversation. If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.
The first round of benchmark tests Visa conducted involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a TV, laptop, or personal care appliance. The results included:
- Across testing, selected AI models achieved an overall benchmark pass rate of 57.2%. The best performing models passed 82.2% of attempts, the worst 28.9%.
- The five AI models that uncovered the most hidden buying factors using conversation flows with the consumer averaged a 70.2% pass rate. The five that uncovered the least passed less than half (44.4%) of the time.
- The five models with the highest average cost per run (standard set of shopping-related operations) passed 67.1% of runs, compared with 50.2% for the five least expensive.
- Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2% of the time.
- More than half (54%) of failed runs still ended in the AI making a purchase. Either the AI recommended a purchase that did not meet the customer’s identified requirements, or it failed to uncover all the customer’s requirements.
As an example of performance testing and results, in one benchmark scenario the customer asked for help choosing a new TV. They mentioned their relative has recommended a brand of television and that they hold a membership with a third-party supplier but did not provide other details.
Through conversation, the AI had to uncover the customer’s budget and additional requirements, such as format compatibility and gaming capabilities. Models that were better at uncovering the customer’s requirements tended to pass more often. But when made to repeat a scenario they had already passed, the same models often failed, which Visa says is indicative of the difficulty of delivering a consistent consumer experience.
In April, the “Business-to-AI (B2AI) Report” revealed that 58% of surveyed U.S. consumers are comfortable with AI agents comparing prices. In addition, 55% of respondents were comfortable with AI applying discounts.
[READ MORE: Visa: Consumers, businesses grow in acceptance of AI agents]
However, only 38% of respondents were comfortable with AI completing a purchase and a lower 27% were comfortable allowing AI to spend money autonomously without limits. Six-in-10 would not allow AI to spend any amount without approval.
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, global head of growth at Visa. “Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry.”
Visa plans to add other shopping scenarios and product categories to Visa Agentic Commerce Benchmark and will extend testing to checkout, payment, and order placement. Visa has shared the benchmark with leading AI model providers for feedback.
