DeepSeek Open-Sources DeepInfer, the Inference Engine Behind Its API
DeepSeek has published DeepInfer, the inference framework that serves its production API, under Apache 2.0 on GitHub. Built primarily on vLLM, it runs DeepSeek-V4.1-Flash on developers' own infrastructure with day-one support for Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. Offline inference and core serving features are included; the KV cache system, prefill and decode disaggregation, and load balancing that complete the production stack are not.
On this page
What DeepSeek actually shipped
DeepInfer is the framework that has been running DeepSeek's commercial API all along. The repository describes it as "the high-performance inference framework that powers DeepSeek's large-scale production services," and the open-source release "provides offline inference with the core features needed for serving at scale." On X, DeepSeek put it more plainly: "Introducing DeepInfer, our in-house inference framework, now fully open source!" and "We're sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable."
The engine started life as a deeply modified fork of vLLM, with the acknowledgments crediting SGLang and TensorRT-LLM as additional inspiration. What ships today is a serving stack tuned for DeepSeek's own architectures: optimized kernels including FlashMLA, DeepGEMM, DeepEP and TileLang, attention and MoE parallelism schemes, an asynchronous scheduler the README calls zero-overhead, structured output support for regex and JSON schemas, and a speculative-style decoding setup built around DSpark and confidence-scheduled verification. The New Stack notes DeepSeek also updated deepseek-recipe, a Rust and Python companion library that maps API requests to model inputs and outputs, first released quietly in September.
Nvidia and Huawei, with equal billing
The hardware support list is the geopolitically loaded part. Day-one support covers Nvidia SM100 data center GPUs and Huawei's Ascend 950 NPU series, with Huawei listed alongside Nvidia rather than as an afterthought. Older Nvidia architectures such as Hopper are absent.
That split reflects where DeepSeek is actually buying silicon. Bloomberg has reported DeepSeek plans at least 160,000 Ascend 950DT chips for a new Inner Mongolia data center, and Reuters has reported collaboration between DeepSeek and Huawei on Ascend programming tools. The parity has limits for now: batch-invariant inference and deterministic sampling are Nvidia-only, and DeepSeek's published benchmarks were run on Nvidia hardware, four SM100s for prefill and 64 for decoding, so real-world Ascend performance remains an open question.
The parts that stay internal
This is a production stack with the production peeled out of it. The README is explicit that a complete service needs three things DeepInfer does not include: a KV cache system, infrastructure for separating prefill and decode across machines, and load balancing across data-parallel replicas. Open-source engineers who have wanted to study how a frontier API is actually served will find the serving engine here, not the orchestration layer around it.
Even so, the release is notable for what it admits. DeepSeek had previously said that open-sourcing this code was not practical, and The New Stack frames the reversal as a deliberate choice to share the production stack. Anyone serving DeepSeek-V4.1-Flash, whose open checkpoint DeepSeek-V4.1-Flash-1017 is on Hugging Face, now gets the same inference engine DeepSeek uses, under an Apache 2.0 license that permits commercial use.
What it means for self-hosting
For readers who run models on their own hardware, the immediate relevance depends on scale. DeepInfer targets multi-GPU and multi-node serving of a large mixture-of-experts model, which is a different world from a phone or a single workstation. But the direction matters: a major model vendor has open-sourced the exact code path between its weights and its paying customers, and made it run on two incompatible chip ecosystems.
The vLLM lineage is the practical headline. Most self-hosting stacks already speak vLLM's dialect, so kernels and scheduling ideas from DeepInfer are likely to migrate into the tools people actually use. And the Huawei support is a quiet bet on a future where Chinese-built inference hardware is a normal part of the open-source stack, on equal terms with Nvidia rather than as a compromised fallback.