How to run DeepSeek-V3.2 locally

Select your machine for an accelerator-specific command. This first iteration uses llama.cpp and a revision-pinned GGUF artifact.

Model benchmarks

Choose your hardware

Showing 4 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Recommended backend

Apple Metal

Calculated macOS

Artifact

Q4_K_M

Estimated GGUF · unsloth/DeepSeek-V3.2-GGUF

Memory required

412.1 GB

Calculated of 435.2 GB usable

Fit at 8K context

Borderline

~31.4–45.2 t/s decode

Verification status: artifact existence checked against Hugging Face on 2026-09-11; command generated from a unit-checked backend template; not executed on physical Mac Studio M5 Ultra 512 GB hardware.

Community conversion: this runnable GGUF is published by unsloth, not by the model publisher. Its exact repository and immutable revision are shown below.

Quick start · llama.cpp (native)Not hardware tested

1. Install llama.cpp

brew install llama.cpp

2. Start a local chat

llama-cli -hf "unsloth/DeepSeek-V3.2-GGUF:Q4_K_M" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

llama-server -hf "unsloth/DeepSeek-V3.2-GGUF:Q4_K_M" -c 8192 -ngl 99 --host 127.0.0.1 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The Docker commands publish port 8080 on 127.0.0.1 only. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for Mac Studio M5 Ultra 512 GB

A current macOS release on Apple Silicon.
Homebrew, or a current llama.cpp binary installed another way.
Enough free unified memory for the weights, KV cache and macOS.

llama.cpp backend documentation ↗

Why this recipe

Metal support is enabled automatically in standard macOS llama.cpp builds.
Run llama.cpp natively: Docker Desktop cannot pass the Apple GPU through to a Linux container.
Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.

Artifact and assumptions

Repository

unsloth/DeepSeek-V3.2-GGUF Estimated

Pinned revision

a787696863ba Estimated source

Artifact quantization

Q4_K_M Estimated

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.