modelrigs

How to run DeepSeek-V4.1 Flash locally

Select your machine for accelerator-specific, revision-pinned commands using DwarfStar.

Model benchmarks
Choose your hardware

Showing 15 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Runtime choices are complementary. Changing runtime selects a different artifact, memory profile and command set; it does not replace the other method.

Recommended backend
NVIDIA CUDA
Calculated Linux
Artifact
DS4 V4.1 Q2 (SSD)
Estimated GGUF · antirez/ds4
Memory required
93.2 GB
Calculated of 105.0 GB usable
Fit at 8K context
Fits
9.4 t/s decode
Verification status: artifact and download instructions checked against the runtime repository on 2026-09-14; command generated from a unit-checked backend template; not executed on physical NVIDIA DGX Spark 128 GB hardware.
Text-only quick start: this recipe starts a text chat. Using the model's vision capability can require a matching multimodal projector and additional llama.cpp flags.
Quick start · DwarfStarNot hardware tested
1. Install DwarfStar
git clone https://github.com/antirez/ds4.git
cd ds4
git checkout 6e4c285a92a1de11863784b748f196b5d4b88a48
make cuda-spark
./download_model.sh ds41f-q2
2. Start a local chat
./ds4 --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ssd-streaming --ctx 8192

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API
Start the server
./ds4-server --cuda -m gguf/DeepSeek-V4.1-Flash-Q2.gguf --ssd-streaming --ctx 8192
Test it from another terminal
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"deepseek-v4-1-flash","messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'
The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.
Prerequisites for NVIDIA DGX Spark 128 GB
  • A Linux system with the NVIDIA driver, CUDA development toolkit, nvcc and cuBLAS installed.
  • nvidia-smi must see the DGX Spark GPU; stop other inference services before loading the model.
  • A fast local SSD with at least 341 GiB free for the model; the weights cannot run resident on one 128 GB Spark.

DwarfStar backend documentation ↗

Why this recipe
  • The repository and commands are pinned to the same DwarfStar revision as the imported benchmark dataset.
  • The server listens on 127.0.0.1:8000 by default and exposes OpenAI-style, Responses-style and Anthropic-style endpoints.
  • Keep --ssd-streaming enabled on one Spark and start without speculative decoding; DwarfStar sizes the expert cache automatically.
  • This 340.598 GiB artifact cannot run resident on one 128 GB DGX Spark; DwarfStar streams routed experts and the 188.833 GiB Engram tables from a fast local SSD.
  • Automatic cache sizing retained 80.17 GiB of experts in the published single-Spark measurements. Leave it automatic unless you are reproducing a documented explicit-cache test.
  • The published 1K-16K sweep generated about 8.2-9.6 tokens/s without speculative decoding. V4.1 CUDA vision and DSpark are not supported at this revision.
Artifact and assumptions
Repository
antirez/ds4 Estimated
Pinned revision
6e4c285a92a1 Estimated source
Artifact quantization
DS4 V4.1 Q2 (SSD) Estimated
Context in commands ⓘ
8,192 tokens Assumption
Execution mode ⓘ
Fully resident Official spec
Hardware execution
Not tested by ModelRigs Estimated
Operational notes
Memory and context
Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.
Persistent downloads
Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.
Performance
The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.