New AI Scheduler URSA Boosts LLM Inference Efficiency
Researchers developed URSA, an uncertainty-aware scheduler that improves latency and service level agreement attainment for LLMs.
Updated on Oct. 26, 2026 in Artificial Intelligence
Live Poll
Do you believe new AI scheduling technology will make your online services more reliable?
Engineers have introduced URSA, an offline reinforcement learning scheduler designed to manage large language model inference more effectively. By predicting output length distributions and memory needs, the system reduces latency and improves reliability in complex GPU clusters.
Why it matters
Traditional schedulers rely on static heuristics that struggle to handle the unpredictability of request output lengths. URSA overcomes this by learning from historical data to make smarter memory and dispatch decisions without the risks associated with online training.
URSA utilizes a graph neural network to analyze the joint state of request queues, GPU instances, and KV cache occupancy. The system reserves memory at a class-dependent high quantile of predicted footprints to prevent evictions and recomputes.
The players
URSA
This is an uncertainty-aware offline reinforcement learning scheduler designed to optimize LLM inference serving.
Azure
This cloud computing platform provided the historical LLM inference traces used to train and test the scheduler.
Alibaba
This technology company provided the historical GPU-cluster traces used to validate the performance of the new scheduling architecture.
The details
The system learns dispatch policies through distributional offline reinforcement learning using historical request traces from platforms like Azure and Alibaba. This approach allows the scheduler to adapt to the inherent uncertainty of generative AI outputs.
Timeline
October 26, 2026: The peer-reviewed research article detailing the URSA scheduler was published.
The Tech Race
URSA addresses the persistent bottleneck surrounding the KV cache in LLM inference servers. This innovation replaces rigid, static heuristic scheduling with dynamic, learned policies to maintain pace with the rapidly increasing demands of generative AI workloads.
Users will benefit from lower latency and more reliable service when interacting with LLM-powered applications. This scheduler helps developers minimize unexpected errors and performance degradation during peak traffic periods.
The takeaway
Dynamic scheduling represents a critical evolution in making generative AI infrastructure more stable and efficient. Adopting uncertainty-aware models allows companies to provide faster, more consistent responses to users without needing constant human adjustment of static rules.
Further reading
For more on how new models are changing server efficiency, explore our Artificial Intelligence section.
More information
Review the technical findings in the peer-reviewed research article.
Source note: This article includes information reported by Nature.
Live Poll
Do you believe new AI scheduling technology will make your online services more reliable?
