AI Models Score 89-91% on Infectious Disease Exam
A study compared five large language models on 227 pharmacy-course infectious disease questions, finding uneven performance on images and specific topics.
Updated on Nov. 7, 2026 in Language Learning
Researchers evaluated five large language models, including ChatGPT-4.1, ChatGPT-4o-mini, Claude, Copilot, and Gemini, on 227 items from an infectious diseases course in a Doctor of Pharmacy program. Overall accuracy ranged from 89.4% to 91.2%, with ChatGPT-4o-mini scoring highest and Claude and Copilot scoring lowest.
Why it matters
The results suggest that general-purpose AI tools can answer much of a pharmacy curriculum accurately, but their reliability drops noticeably on questions that include pictures. Educators and students relying on these tools for coursework should account for that gap.
This evaluation of 227 infectious diseases exam items found overall accuracy between 89.4% and 91.2%, with picture-based questions answered correctly 69% of the time compared with 91% for text-only items. The study does not explain why image questions were harder for the models or whether results would hold for other courses.
The players
ChatGPT-4o-mini
A smaller model in OpenAI's GPT-4 family, which scored highest overall in this comparison.
Claude
A large language model developed by Anthropic, which scored lowest overall in this comparison alongside Copilot.
Copilot
Microsoft's AI assistant, which scored lowest overall in this comparison alongside Claude.
Gemini
Google's large language model family, one of the five models evaluated in this study.
The details
Items were categorized by topic, Bloom's taxonomy level, and format, and accuracy was summarized with descriptive statistics. Chi-Square tests were used to compare results among the models, and inter-model agreement showed a Cohen's Kappa of 0.497 to 0.613.
Roadmap
Language models have been benchmarked against professional licensing exams such as USMLE Step 1 since 2023, and this study extends that testing to pharmacy coursework. The uneven results across question types show that headline accuracy figures can mask weaknesses on specific formats.
Students using AI tools to study for pharmacy or medical coursework should verify answers on image-based questions, where accuracy was notably lower in this evaluation. Checking model output against course materials remains important before relying on it for exams.
The takeaway
AI tools can handle much of a pharmacy exam accurately, but image-based questions remain a clear weak point. Treat AI answers as a study aid to verify, not a substitute for course content.
Further reading
Explore more on Language Learning for related coverage of how AI tools are used in education.
Source note: This article includes information reported by Nature.
