ChatGPT answers questions about cancer

Artificial intelligence (AI)-powered chatbots can potentially improve healthcare, particularly in answering questions by patients regarding cancer, reveals a recent study.

From February to March 2023, study authors asked chat generative pretrained transformer (ChatGPT) from OpenAI and Bing AI questions from the American Cancer Society. Questions were customized for all stages of breast, colon, lung, and prostate cancer and were grouped by type (ie, prognosis, treatment, or miscellaneous).

An expert panel assessed the quality of AI chatbot responses using the validated DISCERN criteria.

A total of 117 questions were raised to ChatGPT and Bing. The average scores for all questions were 3.9 for ChatGPT and 3.2 for Bing (p<0.001), while the overall DISCERN scores were 4.1 and 4.4, respectively.

By disease site, the corresponding average scores for ChatGPT and Bing were 3.9 and 3.6 for prostate cancer (p=0.02), 3.7 and 3.3 for lung cancer (p<0.001), 4.1 and 2.9 for breast cancer (p<0.001), and 3.8 and 3.0 for colorectal cancer (p<0.001).

By type of question, the average score for ChatGPT and Bing were 3.6 and 3.4 for prognostic questions (p=0.12), 3.9 and 3.1 for treatment questions (p<0.001), and 4.2 and 3.3 for miscellaneous questions (p=0.001), respectively. Notably, at least one panellist rated three responses (3 percent) by ChatGPT and 18 (15 percent) by Bing as “having serious or extensive shortcomings.”

“This analysis suggests a critical need, particularly around cancer prognostication, for continual refinement to limit misleading counselling, confusion, and emotional distress to patients and families,” the authors said.