Visa benchmark finds AI models fail 42.8% of shopping tests
V•Visa’s AI shopping benchmark found that 42.8% of test runs across 22 models failed. Models passed all three attempts at a shopping scenario 34.2% of the time.
1. Benchmark results
Visa’s benchmark tested 22 AI models on shopping scenarios. Purchases still occurred in 54% of failed runs, despite unmet customer requirements or incomplete discovery of customer needs.
2. Performance varied
Pass rates ranged from 28.9% for the worst-performing models to 82.2% for the best. The five most expensive models averaged a 67.1% pass rate, compared with 50.2% for the five least expensive.
