Lila Sciences Trains Model on Million-Token Sequences
Lila Sciences and Anyscale used 64 NVIDIA B200 GPUs to train GLM-5.3-Flash on sequences of up to 1 million tokens.
Updated on Nov. 7, 2026 in Life Sciences
Lila Sciences and Anyscale trained GLM-5.3-Flash on sequences of up to 1 million tokens using 64 NVIDIA B200 GPUs and the open-source SkyRL training stack. The run marked a 64-fold increase over the initial training capacity.
Why it matters
Training on sequences this long requires managing memory and processing demands that limit how much context a model can use at once. The results show a path to longer-context scientific models, including a human-genome fine-tuning test on 512,000-token sequences that reduced held-out loss by 4.8%.
The training benchmarks come from a single reported run on 64 NVIDIA B200 GPUs, with throughput of 13,700 tokens per second at 1 million tokens compared with a peak of 24,900 tokens per second at 256,000 tokens. Whether these results generalize to other hardware or longer sequences has not been established.
The players
Lila Sciences
Lila Sciences is a company that collaborated with Anyscale on training large models for scientific applications.
Anyscale
Anyscale is a company that provides distributed computing infrastructure for running large AI workloads.
The details
The model, GLM-5.3-Flash, combines 34 layers of Kimi Delta Attention with 11 layers of DeepSeek Sparse Attention. Changes to tensor dimensions enabled sparse-attention kernels for the DeepSeek layers and halved memory use for that component, while activation recomputation, CPU offloading and context parallelism managed the remaining demands.
Timeline
The training results were reported on November 7, 2026.
Big Picture
The run relies on the NVIDIA B200 GPU, which it used in a 64-GPU configuration to reach million-token training. Its results add to the push to extend context length in models trained for scientific use.
For researchers working with long genomic or document sequences, the reported techniques for fitting million-token training on current GPUs point to practical options for memory management. The gains are shown in a research setting, so access to comparable hardware remains a limiting factor for most readers.
The takeaway
Long-sequence training is becoming more practical through memory-efficient architectures and offloading techniques. Readers following AI for life sciences can track whether these methods carry over to other biological datasets.
Further reading
For related coverage, explore the Life Sciences section on Wisevoter.
Source note: This article includes information reported by TokenPost.
