ChatGPT-5 missed 83 of 170 distal radius fractures on wrist radiographs when asked a single yes-or-no question, a sensitivity of 51.2%, according to a study published October 26 in Emergency Radiology. The model also called a fracture on 59 of 200 normal studies. The authors, from Rambam Health Care Campus in Haifa, Israel, conclude that a general-purpose chatbot should not be used to decide whether a radiograph shows a fracture.
How ChatGPT-5 was tested on wrist radiographs
Bana Awad, Yaniv Keren, and Dror Ronel drew their images from MURA, a public dataset released by Stanford in 2017 with 40,561 upper-extremity images whose studies were labeled normal or abnormal by radiologists during clinical reads, and widely used to train and benchmark algorithms. Of 3,697 anteroposterior (AP) distal radius views in the archive, they removed poor-quality images, pediatric studies (open physes can mimic a fracture line), and radiographs with orthopedic hardware. Images taken through a cast were kept.
Because the MURA label applies to a whole study and does not specify a fracture, the team set its own reference: a senior orthopedic resident and a board-certified orthopedic surgeon reviewed every image together, before any image reached the model, and included it only when they agreed a fracture was definitely present or definitely absent.
Between December 1 and February 28, each radiograph was uploaded to the public ChatGPT web interface in a fresh temporary chat, with no system prompt, no examples, memory and custom instructions switched off, and default settings. The prompt never changed: “Is there a fracture in this X-ray? Answer with yes or no.” The team made 396 submissions. In 26 of them (6.6%) the model hedged instead of answering, and those images were dropped from the primary analysis, which therefore covered 370 radiographs: 170 with a fracture (45.9%) and 200 normal.
51.2% sensitivity, with errors tilted toward missed fractures
The confusion matrix shows 87 fractures detected and 83 missed, plus 141 normal studies correctly cleared and 59 false alarms. That yields a sensitivity of 51.2% (95% CI 43.7–58.6), specificity of 70.5% (63.8–76.4), and accuracy of 61.6% (56.6–66.4). Positive predictive value was 59.6% and negative predictive value 62.9%. Cohen’s kappa of 0.219 indicates only fair agreement with the reference read. A post hoc McNemar test confirmed that false negatives significantly outnumbered false positives (χ² = 4.06, p = 0.044).
Adding the 26 hedged answers back in moved sensitivity between 47.3%, when they were counted as errors or as “no,” and 54.9%, when treated as “yes,” the assumption most generous to the model. Projected to a department with 20% fracture prevalence, the model would miss 98 fractures and raise 236 false alarms per 1,000 radiographs, for a positive predictive value of 30.3%.
“Put plainly: a general-purpose chatbot should not be used to decide whether a radiograph shows a fracture,” the researchers wrote.
Why a multimodal LLM is not a fracture detector
The obvious benchmark is dedicated fracture-detection software built on convolutional neural networks (CNNs), trained end to end on labeled radiographs for one job. The first such product authorized by the FDA, Imagen’s OsteoDetect, received De Novo marketing authorization in May 2018 specifically to flag distal radius fractures in adults, as an aid to, not a replacement for, clinician review. Meta-analyses cited in the paper put pooled sensitivity for this class of tool at around 90%. The authors stress that the comparison is indirect, since populations, datasets, and reference standards differ, but argue that the gap is too large to blame on methodology alone.
The paper lists plausible mechanisms that its design cannot tease apart. A multimodal model’s visual encoder is tuned for natural images, not for faint grayscale cortical breaks. The binary prompt rules out the step-by-step reasoning and clinical context that, in other studies, lifted these models’ accuracy by 5 to 15 percentage points, which makes this result a floor rather than a ceiling. And the consumer interface resizes and tiles images before the model sees them, in ways it does not disclose.
Why missed fractures are the dangerous error in triage
In triage, which way a tool errs matters as much as how often. A false positive costs a second look, an extra view, or an unnecessary orthopedic referral. A false negative can send a patient home with an unprotected fracture and a risk of displacement and malunion. A system that says “no” to half of all fractures shifts risk onto exactly the side triage is meant to protect. The scenario the authors set out to reproduce is a non-radiologist at the point of care who uploads an image and asks the simplest possible question. They also note that about one query in fifteen returned no usable answer at all.
The same pattern shows up in other recent work. Even purpose-built tools stumble without clinical context, as in the case of the AI that flagged a pneumothorax that was not there, and seven commercial chest X-ray algorithms for lung cancer ranged from 21% to 78% in sensitivity. When an LLM argues for a wrong answer, it is radiologist expertise that keeps the error from being accepted, a safeguard a non-specialist user lacks. Text tasks under supervision are a different matter: GPT-5 has been used to help patients understand their imaging reports, and an LLM raised trainee accuracy on brain MRI as a reasoning aid rather than the final judge.
On the regulatory side, the paper notes that ChatGPT-5 is not cleared or approved as a medical device for radiological use by the FDA, the EMA, the UK MHRA, or Israel’s Ministry of Health.
Study limits and what to watch next
The authors scope their finding tightly: it applies to this model version, this prompt, this anatomical region, and this dataset, and does not settle whether carefully engineered LLM workflows could contribute to musculoskeletal imaging. The reference came from two orthopedists reading together rather than radiologists reading independently, with no CT or follow-up confirmation. Because only radiographs both readers judged clear-cut made it in, the numbers likely represent the model’s best case. Each image was submitted once to a non-deterministic service that is updated continuously, and alternative prompts, clinical history, and other models were not explored.
For the field to move forward, the group calls for task-specific adaptation, rigorous prompt and pipeline design, head-to-head comparison with dedicated CNNs on the same images, and testing across several anatomical regions with radiologist-adjudicated reference standards. Until then, the practical reading is that validated fracture detectors and a conversational chatbot are different technologies backed by different evidence, and, as the authors put it, “treating them as one would misrepresent both.”
Source: AuntMinnie — ChatGPT-5 misses half of distal radius fractures on x-rays. Original paper (open access): Awad B, Keren Y, Ronel D. Performance of ChatGPT-5 under a minimal binary prompt for detection of distal radius fractures on plain radiographs. Emergency Radiology, 2026.
