preprint Open access
Papers open inside SciRah from the original publisher or repository; SciRah never hosts them.
Abstract Background General-purpose large language models (LLMs) are increasingly evaluated in diagnostic pathology, but prior studies have largely emphasized diagnostic accuracy rather than how models fail. We evaluated four LLMs for diagnostic performance, pathology-relevant errors and hallucinations, their burden, and potential clinical impact across multi-organ pathology cases. Design In this retrospective multicenter study, 153 pathology cases from two institutions spanning 20 organs were evaluated using ChatGPT-5.3 (LLM1), Gemini 3 (LLM2), Grok 4.20 (LLM3), and Claude Opus 4.6 (LLM4). Each LLM received multi-magnification histologic images with clinical context and generated a microscopic description and diagnosis. No data splitting, training, or fine-tuning was performed. Twenty-one pathologists assessed 612 outputs for diagnostic correctness, error and hallucination type and burden, clinical impact (0–4), and overall performance (1–5). Results Strict diagnostic accuracy was 48.9% overall (60.1% including partially correct diagnoses) and ranged from 36.6% to 58.2% across LLMs. Errors occurred in 82.0% of outputs and hallucinations in 76.6%; 90.2% contained at least one error or hallucination. Misinterpretation was the most frequent error (72.1%), while fabricated histologic features were the dominant hallucination type (74.5%). LLM2 had significantly lower misinterpretation rates than the other three LLMs, while LLM3 had significantly higher fabricated-feature hallucination rates than LLM1 and LLM2. Errors were more frequent in incorrect than correct diagnoses (97.5% vs 66.2%), as were hallucinations (95.1% vs 59.9%; both p < 0.001). Every strictly incorrect diagnosis contained an error and/or hallucination, while 79.9% of strictly correct diagnoses also contained at least one. In multivariable analysis, LLM3 was independently associated with higher odds of an incorrect diagnosis. Diagnostic correctness was the strongest determinant of high clinical impact; each one-point increase in hallucination burden increased the odds by 63% (adjusted OR, 1.63). Conclusion Diagnostic accuracy alone substantially underestimates the safety limitations of general-purpose LLMs in histopathology. Errors and hallucinations were common even when the final diagnosis was correct, and their burden was independently associated with clinical impact. Evaluation frameworks should therefore assess both diagnostic correctness and the reliability and potential consequences of accompanying generated content.
TL;DRDiagnostic accuracy alone substantially underestimates the safety limitations of general-purpose large language models in histopathology, and evaluation frameworks should assess both diagnostic correctness and the reliability and potential consequences of accompanying generated content. (Semantic Scholar)