Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making

This study asked whether three large language models (ChatGPT, Claude, and DeepSeek) make better endodontic decisions for patients taking antiresorptive drugs, where tooth preservation must be weighed against the risk of medication-related osteonecrosis of the jaw. The authors also tested whether giving the models the 2022 American Association of Oral and Maxillofacial Surgeons position paper improved their answers. The researchers wrote 18 fictional, standardized clinical scenarios based on that guideline and current evidence. Each model answered them twice: once without help, and once after receiving the full guideline. Two endodontic specialists scored the anonymized responses across five decision-making domains on a 0 to 5 scale, and their disagreements were resolved by consensus before analysis. Statistical analysis used linear mixed models, with ordinal models as a sensitivity check. At baseline, ChatGPT scored higher than Claude (4.68 vs. 4.29), while DeepSeek scored lowest. After guideline-assisted prompting, Claude rose to 4.68, a gain of 0.39 points, and the difference between ChatGPT and Claude was no longer significant. The largest model-specific gain was in follow-up recommendations. A significant phase effect and model-by-phase interaction were found, though one domain's phase effect was not confirmed by the sensitivity analysis. Because the scenarios, reference standard, and prompt all came from the same guideline, the scores measure agreement with that guideline rather than independent clinical accuracy. The abstract does not describe testing on external scenarios or real patient cases, so clinical use would require that further work.