AI Is Really Bad at Summarizing the News. Should We Be Worried?
By Movieguide® Contributor
The BBC recently put four of the most popular AI chatbots to the test and found that all of them perform quite underwhelmingly when asked to accurately summarize news articles.
During the study, researchers fed OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini and Perplexity AI 100 news articles and asked for an accurate summary. The responses were disappointing as 51% of all answers included significant issues, and 19% introduced factual errors, such as making up facts, names and dates.
According to The Verge, some of the examples of inaccuracies included Gemini saying that the UK’s National Health Service “advises people not to start vaping, and recommends that smokers who want to quit should use other methods.” But the NHS actually does recommend the practice for smokers who want to quit. Another error? ChatGPT said that Ismail Haniyeh was still part of Hamas leadership in December 2024, even though he was assassinated in July 2024.
Furthermore, these chatbots struggled to differentiate between opinion pieces, editorials and news stories, oftentimes leaving out crucial information.
“The price of AI’s extraordinary benefits must not be a world where people searching for answers are served distorted, defective content that presents itself as fact,” BBC News and Current Affairs CEO Deborah Turness wrote. “In what can feel like a chaotic world, it surely cannot be right that consumers seeking clarity are met with yet more confusion.”
Despite these troubling results, OpenAI is adamant that its chatbot is a helpful tool for those hoping to keep up with current events, and it also helps news websites by promoting their stories.
“We support publishers and creators by helping 300 million weekly ChatGPT users discover quality content through summaries, quotes, clear links and attribution,” an OpenAI spokesperson told the BBC.
“We’ve collaborated with partners to improve in-line citation accuracy and respect publisher preferences, including enabling how they appear in search by managing OAI-SearchBot in their robots.txt,” the company later added. “We’ll keep enhancing search results.”
While OpenAI asserts that its chatbot is an overall useful tool, Apple has acknowledged that AI might not yet be able to be used in this field. Last month, Apple paused its AI from summarizing news stories after a string of absurdly bad summaries.
READ MORE: APPLE PAUSES AI NEWS ALERTS AFTER GENERATING FALSE REPORTS
AI’s struggle with accurate summaries points to a larger problem with the technology where it “hallucinates” data and information to provide an answer. This is particularly common when the technology is tasked with a prompt that it does not have the relevant information to sufficiently answer. However, it has been trained to always provide a response, causing it to make things up rather than simply replying “I don’t know.”
While many users are starting to turn to AI for their news, the majority of people still turn to an even more surprising place to keep up to date: social media.
READ MORE: THE SURPRISING PLACE 17% OF AMERICANS GET THEIR NEWS
- Content: +3Share:
- Content: +1