DeepSeek Open Sources DeepInfer: What Its Inference Engine Means for AI Costs
On Tuesday, October 20, 2026, DeepSeek put the software it uses to serve its own models out in the open. It is called DeepInfer, and the GitHub README describes it as “the high-performance inference framework that powers DeepSeek’s large-scale production services.” The license is Apache 2.0.
I like stories like this more than another benchmark chart. Most of the money in AI right now goes into running models, not training them, and the serving layer is usually the part companies keep to themselves. So when a lab shares that layer, I want to understand what is actually in the box and who can use it.
Here is my plain read of the release, what it can and can’t do, and how I’d approach it.
What DeepSeek actually released
DeepInfer is an inference engine. Inference is the work of running a trained model to answer requests, and the same model can be cheap or expensive to run depending on the software around it.
According to the README, this first public release supports one model, DeepSeek V4.1 Flash, and two kinds of hardware: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. The default checkpoint it pulls is DeepSeek V4.1 Flash 1017 on Hugging Face.
The Next Web reported on October 20 that DeepSeek announced it on X that same day, saying it was sharing its production stack “to help the community lower compute costs, reduce energy use, and make AI more affordable.” The same report says DeepSeek also released a companion called deepseek recipe that matches how its official API turns text into tokens and back.
The repo moved fast. When I checked the GitHub API on the morning of October 21 IST, it showed more than 13,500 stars and about 1,260 forks. The Next Web had counted more than 8,000 stars by Tuesday afternoon. Stars tell you about attention, though, not quality.
How DeepInfer works, in simple terms
The README lists a handful of features. Here is what each one means in practice, as far as I can tell from DeepSeek’s own description.
- Optimized kernels. Low level code tuned for the hardware, built on DeepSeek’s earlier open projects like FlashMLA, DeepGEMM and DeepEP.
- Parallelism. Ways to split attention and the mixture of experts layers across many chips, with load balancing between experts.
- DSpark. A speedup trick where the engine guesses several tokens ahead, then uses confidence scores to decide when to check those guesses. The README says the default maximum guess length is 5 tokens.
- Asynchronous scheduling. The CPU plans the next batch while the accelerator is still busy with the current one. This is on by default.
- Batch invariant inference and deterministic sampling. Same input and settings give the same output, no matter what else is in the batch. The README marks both as NVIDIA only.
- Structured outputs. Forcing answers into a regex, JSON schema or grammar.
That last pair caught my eye. Anyone who has tried to debug an agent knows how painful it is when the same prompt gives a different answer each run. I wrote about that pain in my LLM observability guide, and repeatable outputs make tracing and evals far easier.
DeepSeek also credits its roots. The README says DeepInfer “was originally built primarily on vLLM,” with ideas from SGLang and NVIDIA’s TensorRT LLM. The Next Web adds that the vLLM project thanked DeepSeek and said it would bring ideas from DeepInfer into its own V4.1 Flash support.
What is missing from the box
This part matters if you are thinking of running it yourself. The README is clear that the release is for offline inference and leaves out three things you need for a complete service:
- A KV cache system
- Splitting prompt processing and answer generation onto separate machines (prefill and decode disaggregation)
- Load balancing across replicas for both of those stages
So this is not a server you install on Monday and put behind your app on Tuesday. It is the engine, without the plumbing around it. Teams that already run large serving stacks can plug it into what they have. Smaller teams will find it more useful as a reference for how DeepSeek does things.
The prerequisites are also specific: Linux, GCC 13 or newer, Python 3.12, and either CUDA 13 on NVIDIA or CANN 9.2 on Ascend, according to the README and Quartz’s October 20 report. SM100 is NVIDIA’s current data center class, so this is not something you’ll run on a gaming card.
The speed claims are DeepSeek’s own
The README includes throughput and latency charts, measured on 4 SM100 GPUs for prefill and 64 for decode. It also ships a script so others can reproduce the sweep.
I’d treat those charts as vendor claims for now. The Next Web notes that the figures are DeepSeek’s own and, citing Bloomberg, that nobody outside the company had independently benchmarked the gains yet. Once people with the right hardware run that sweep, we’ll know how much of the gain holds up outside DeepSeek’s own cluster.
Why it matters for AI costs
The cost angle is the real story. TechTarget’s October 20 report frames the release around token bills, since agents burn through huge numbers of tokens. Omdia analyst Lian Jye Su told TechTarget that “token consumption has been going through the roof,” and that turning tasks around quickly and cheaply helps companies get more out of their AI infrastructure.
Who benefits first is fairly narrow. Cloud and hosting firms already serving DeepSeek models can now use the same methods DeepSeek uses, and companies running those models on their own servers may cut costs too, The Next Web reported, citing Bloomberg. If you use a different model, TechTarget points out you already have other open engines like TensorRT LLM.
The bigger, slower effect is on the open tools everyone uses. If vLLM and SGLang pick up these ideas, people running all sorts of models could see the gains later without ever touching DeepInfer.
The politics you should know about
There is context here that I won’t skip. The Next Web reported that the release came a day after the US ordered civilian federal agencies to remove DeepSeek products from their systems, and TechTarget mentions the same federal ban.
The code itself is Apache 2.0 and readable, which is more than you get with most serving stacks. But if you work in a regulated field or sell to the US government, check your own rules on DeepSeek software and models before anyone installs it. That is a policy question for your company, not a technical one.
The Huawei support is also worth noting. TechTarget’s analysts read the dual NVIDIA and Ascend support as good news for Huawei as it tries to look like a serious alternative, while also showing that NVIDIA chips still matter for DeepSeek.
What I would do this week
If you run DeepSeek models in production on SM100 or Ascend 950 hardware, this is worth an afternoon. Build it in a clean Python 3.12 environment, run the included offline example, then run the benchmark sweep on your own hardware and compare it with what you use today. Trust your own numbers over anyone’s chart, including DeepSeek’s.
If you don’t have that hardware, which is most of us, read the README’s feature list and follow what vLLM and SGLang do with these ideas in their next releases. That’s where most people will actually feel this change.
And if your company has rules on Chinese AI vendors, raise this with whoever owns that policy before you clone anything.
By Deepu
Leave a comment