How to run gpt-oss-20b locally

Select your machine for an accelerator-specific command. This first iteration uses llama.cpp and a revision-pinned GGUF artifact.

Model benchmarks

Choose your hardware

Machine

Showing 25 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Recommended backend

NVIDIA CUDA

Calculated Linux

Artifact

MXFP4

Estimated GGUF · ggml-org/gpt-oss-20b-GGUF

Memory required

15.8 GB

Calculated of 992.6 GB usable

Fit at 8K context

Comfortable

~553–796 t/s decode

Verification status: artifact existence checked against Hugging Face on 2026-09-11; command generated from a unit-checked backend template; not executed on physical NVIDIA DGX H200 (8x H200, 1,128 GB) hardware.

Quick start · llama.cpp (CUDA Docker image)Not hardware tested

1. Start a local chat

docker run --rm -it \
  --gpus all \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full-cuda -hf "ggml-org/gpt-oss-20b-GGUF:MXFP4" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

docker run --rm \
  --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:server-cuda -hf "ggml-org/gpt-oss-20b-GGUF:MXFP4" -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The Docker commands publish port 8080 on 127.0.0.1 only. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for NVIDIA DGX H200 (8x H200, 1,128 GB)

A working NVIDIA driver on the host.
Docker with NVIDIA Container Toolkit configured; `docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi` should see the GPU.
Enough aggregate VRAM for the selected model. Multi-GPU machines also depend on llama.cpp's split and interconnect behaviour.

llama.cpp backend documentation ↗

Why this recipe

The `--gpus all` flag exposes NVIDIA GPUs to the container; the generic llama.cpp image would otherwise run on the CPU.
The named Docker volume keeps the downloaded model between runs.
MXFP4 is this model's native reference format rather than a post-training Q4 conversion.
Use a current llama.cpp build so the model architecture and bundled chat template are supported.

Artifact and assumptions

Repository

ggml-org/gpt-oss-20b-GGUF Estimated

Pinned revision

ef9b12f2ff56 Estimated source

Artifact quantization

MXFP4 Estimated

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.