Language Models Vary in Bladder Cancer Recommendations

A retrospective study of nine large language models found physician ratings of their MRI-based bladder cancer recommendations depended on input language.

Updated on Nov. 13, 2026 in Artificial Intelligence

Bold flat-color editorial illustration of three glass blocks of uneven height holding layered liquids beside an open circular frame, in deep red, cream, and teal.
Researchers found that nine large language models gave bladder cancer treatment recommendations that physicians rated differently depending on whether MRI reports were translated into English. AI Illustration. Upload story photo >

Researchers evaluated nine API-accessed large language models generating bladder cancer management recommendations from 590 multiparametric MRI reports drawn from three hospitals. The study was published in npj Digital Medicine as an accepted early version on November 13, 2026.

Why it matters

The findings suggest that translating clinical reports into English can change how physicians rate model output, but the effect differed across models rather than applying uniformly. Clinicians and developers evaluating these tools for oncology decisions may need to test each model under the actual language conditions of their practice.

The evaluation rests on physician ratings from 590 MRI reports tested in both Chinese and clinician-translated English, with inter-rater agreement substantial (quadratic weighted kappa 0.621-0.743). The study does not report whether these ratings predict clinical outcomes, and the cause of the model-specific language effect remains unexplained.

The details

Two physicians independently rated each model recommendation on a 1-5 scale, and analysis used patient-level random-intercept cumulative link mixed models, with paired Wilcoxon tests as secondary exploratory analyses. The likelihood-ratio chi-square for the input-condition interaction was 683.05, indicating the effect of language was not the same across the nine models.

Timeline

  1. November 13, 2026: Article published in npj Digital Medicine as an accepted early version.

Big Picture

The study appears through npj Digital Medicine's accepted early-version pathway, which makes peer-reviewed digital health AI evaluations available before final typesetting. It adds to a growing set of head-to-head comparisons of clinical language models, extending that work to multiparametric MRI reporting for bladder cancer.

Patients receiving bladder cancer care are unlikely to see direct effects from this study, which evaluated model outputs rather than deployed clinical tools. Clinicians considering such tools should note that the language of the input report can alter how physicians judge the recommendations.

The takeaway

Language of input can change how physicians rate large language model recommendations, and that effect differs by model. Verifying a tool under the exact input conditions used in practice is a sensible step before relying on it.

Further reading

Explore more coverage in the Artificial Intelligence section.

Source note: This article includes information reported by Nature.