Together AI Launches Compiler for AI Inference
The new compiler uses AI agents to rewrite GPU code and cuts engine setup time from 467 seconds to nine seconds.
Updated on Nov. 5, 2026 in Artificial Intelligence
Together AI introduced a compiler for AI inference workloads that reduces engine setup time from 467 seconds to nine seconds. The compiler reached 252 times the experimentation throughput of baseline agentic systems and is operational.
Why it matters
Faster engine setup and higher throughput could lower the time and cost required to run AI models at scale. The compiler's use of agents to rewrite GPU code points to automated optimization becoming part of inference infrastructure.
The compiler reached 1,111 tokens per second on MiniMax M3.1-Flash and 1,205 tokens per second on DeepSeek v4.1-Flash on the InferenceX low-latency benchmark. It outperformed open-source megakernels by 1.7 times in a Llama 1B test with a batch size of one, though the source does not report independent verification of these results.
The players
Together AI
Together AI is an AI infrastructure company that provides cloud platforms for running and fine-tuning open-source and custom AI models.
The details
The compiler uses agents to rewrite GPU code, then tests candidate changes with a two-tier system of fast and slower engines. It allows reloading of candidate code changes without restarting and selects parallelism strategies using hardware specifications, microbenchmarks, and real-world traffic data.
Timeline
Nov. 4, 2026: Together AI introduced the inference compiler.
Less than 24 hours: Throughput tests completed across Kimi K3, GLM 5.3, MiniMax M3.1-Flash, and DeepSeek v4.1-Flash.
The Tech Race
The compiler automates the hand-tuned GPU kernel work that has long been done against NVIDIA's CUDA programming model. Its agent-driven rewriting places Together AI in a growing contest to make AI inference faster on existing hardware.
Developers running AI models could see faster startup times and higher throughput if the compiler performs as reported. Teams evaluating inference platforms may compare these benchmark claims against their own workloads before adopting new tooling.
The takeaway
Faster inference tooling can reduce the wait and cost of running AI models. Benchmark gains from a single company's tests are worth checking against independent results before drawing broad conclusions.
Further reading
Explore more on Artificial Intelligence developments shaping how models run in production.
Source note: This article includes information reported by TokenPost.
