---
source_url: "https://arxiv.org/abs/2605.22785?utm_source=openai"
title: "[2605.22785] Evaluating Commercial AI Chatbots as News Intermediaries"
mirrored_at: 2026-08-14T03:33:32.624Z
host: arxiv.org
cited_in_42a: true
mirror_canonical: "https://index.42a.ai/arxiv.org/abs/2605.22785__q__utm_source_openai"
---

> **Original source:** https://arxiv.org/abs/2605.22785?utm_source=openai

[View PDF](https://arxiv.org/pdf/2605.22785) [HTML (experimental)](https://arxiv.org/html/2605.22785v1)

> Abstract:AI chatbots are rapidly shaping how people encounter the news, yet no prior study has systematically measured how accurately these systems, with their proprietary search integrations and retrieval-synthesis pipelines, handle emerging facts across languages and regions. We present a 14-day (February 9-22, 2026) evaluation of six AI chatbots (Gemini 3 Flash and Pro, Grok 4, Claude 4.5 Sonnet, GPT-5 and GPT-4o mini) on 2,100 factual questions derived from same-day BBC News reporting across six regional services (US & Canada, Arabic, Afrique, Hindi, Russian, Turkish). The best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier. The same systems, however, lose 11-13% under free-response evaluation, and 16-17% across the cohort. We further characterize three failure patterns. First, every model achieves its lowest accuracy on Hindi (79% vs. 89-91% elsewhere) and citations indicate an Anglophone retrieval bias (e.g., models answering Hindi queries cite English Wikipedia more than any Hindi outlet). Second, retrieval, not reasoning, failures drive over 70% of all errors. When models retrieve a correct source, they often extract the correct answer; the problem is to land on the right source in the first place. Third, models achieving 88-96% accuracy on well-formed questions drop to 19-70% when questions contain subtle false premises, with the most vulnerable model accepting fabricated facts 64% of the time. We also identify a detection-accuracy paradox: the best false-premise detector ranks second in adversarial accuracy (abstention rate), while a weaker detector ranks first, showing that premise detection and answer recovery are partially independent capabilities. Overall, these suggest that high accuracy can mask systematic regional inequity, near-total dependence on retrieval infrastructure, and vulnerability to imperfect queries real users pose.

Comments:

[this https URL](https://suzgunmirac.github.io/ai-news-preview/)

Subjects:

Computation and Language (cs.CL)

Cite as:

[arXiv:2605.22785](https://arxiv.org/abs/2605.22785) \[cs.CL\]

 

(or [arXiv:2605.22785v1](https://arxiv.org/abs/2605.22785v1) \[cs.CL\] for this version)

 

[https://doi.org/10.48550/arXiv.2605.22785](https://doi.org/10.48550/arXiv.2605.22785)

arXiv-issued DOI via DataCite

## Submission history

From: Mirac Suzgun \[[view email](https://arxiv.org/show-email/139156bd/2605.22785)\]  
**\[v1\]** Thu, 21 May 2026 17:42:07 UTC (1,710 KB)