When you need accurate information about a serious illness, should you go to Google or ChatGPT?
The study, published in the Journal of Medical Internet Research, showed that Google provides the most current information, but query results are skewed by service and product providers seeking customers.
ChatGPT, meanwhile, provides more objective information, but it can be outdated and lacks the sources of its information in its narrative responses, the researchers found.
"If you pick the best features of both, you can build a better system, and I think that this is what will happen in the next couple of years," said Vagelis Hristidis, a professor of computer science and engineering in University of California, Riverside’s Bourns College of Engineering.
A team of scientists submitted 60 queries to both Google and OpenAI’s ChatGPT that are typical submissions from people living with dementia and their families. The researchers said they focused on dementia because more than 6 million Americans are impacted by Alzheimer's disease or a related condition and caregivers of people living with dementia are among the most engaged stakeholders in pursuing health information.
Half of the queries submitted by the researchers sought information about the disease processes, while the other half sought information on services that could assist patients and their families.
The results were mixed.
"Google has more up-to-date information, and covers everything," Hristidis said.
"Whereas ChatGPT is trained every few months. So, it is behind. Let's say there's some new medicine that just came out last week, you will not find it on ChatGPT."
While dated, ChatGPT provided more reliable and accurate information than Google. This is because the ChatGPT creators at OpenAI choose the most reliable websites when they train ChatGPT through computationally intensive machine learning. Yet, users are left in the dark about specific sources of information because the resulting narratives are void of references.
Google, however, has a reliability problem because it essentially "covers everything from the reliable sources to advertisements", Hristidis said. In fact, advertisers pay Google for their website links to appear at the top of search result pages. So, users often first see links to websites of for-profit companies trying to sell them care-related services and products.
Finding reliable information from Google searches thus requires a level of user skill and experience, Hristidis said.
Further, Google and ChatGPT both scored low for readability scores, which makes it difficult for people with lower levels of education and low health literacy skills.