Benchmark Finds Single-Cell AI Models Beat Baselines
A new benchmark tests five single-cell foundation models on cell annotation, perturbation prediction, and regulatory inference against simple baselines.
Updated on Nov. 4, 2026 in Artificial Intelligence
Researchers evaluated scGPT, Geneformer V1 and V2, Nicheformer, and scFoundation alongside CellOracle, supervised perturbation models, and simple baselines. The benchmark was published on bioRxiv on November 4, 2026 and released as the fmaudit package.
Why it matters
Single-cell foundation models are increasingly proposed for biology and drug discovery, and a standardized comparison against simple baselines helps clarify where they add value. The results show the models perform well on some tasks while simple methods remain competitive on others.
Nicheformer reached 0.98 accuracy on human-to-mouse lung label transfer, while Geneformer V2 and scFoundation reached about 0.90 and the best baseline reached 0.68. Up to 75% of perturbations shifted in the expected direction in K562 zero-shot in silico predictions using scFoundation, though the extraction does not address generalization beyond these tasks.
The details
The benchmark used K562 Perturb-seq screens, primary cells, healthy-donor and isogenic leukodystrophy brain organoids, a public organoid CRISPR screen, and lung cancer biopsies. Within a species, foundation model embeddings annotated cells as accurately as principal components, and predicted knockout effects in organoids were largely explained by target gene expression location.
Timeline
November 3, 2026: Core benchmark results were reported.
November 4, 2026: Benchmark preprint published on bioRxiv.
The Tech Race
The benchmark tests whether single-cell foundation models built on large pretraining corpora outperform simple methods on K562 Perturb-seq screens and related datasets. Its comparison against CellOracle and supervised perturbation models places these tools within an ongoing effort to establish which computational approaches genuinely advance cell-level prediction.
Researchers designing biology experiments may use the released fmaudit package to check whether a foundation model beats simple baselines on their own data. The benchmark's results suggest that choosing a model for cell annotation or perturbation prediction should involve comparison against baselines rather than assuming foundation models will outperform them.
The takeaway
Simple baselines remain competitive on several tasks, so testing a single-cell foundation model against a basic method on your own dataset is a practical step before adopting it. Standardized benchmarks like this one make those comparisons easier to run.
Further reading
Explore more on Artificial Intelligence developments and how these models are being evaluated.
