How to run Nemotron 3.5 Lightning 30B-A3B locally

Select your machine for accelerator-specific, revision-pinned commands using llama.cpp.

Choose your hardware

Machine

Showing 24 compatible configurations at 8K context with extra headroom for the Q4_0 artifact. Selection changes the instructions, not the canonical URL.

Recommended backend

CPU

Calculated Linux, macOS or Windows with Docker

Artifact

Q4_0

Estimated GGUF · ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF

Memory required

20.0 GB

Calculated of 176.6 GB usable

Fit at 8K context

Comfortable

~16.9–24.3 t/s decode

Verification status: artifact and download instructions checked against the runtime repository on 2026-09-15; command generated from a unit-checked backend template; not executed on physical CPU-only workstation (Ryzen 9950X, 192 GB DDR5) hardware.

Quick start · llama.cpp (CPU Docker image)Not hardware tested

1. Start a local chat

docker run --rm -it \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full -hf "ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0" -c 8192 -t -1 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

docker run --rm \
  -p 127.0.0.1:8080:8080 \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:server -hf "ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0" -c 8192 -t -1 --host 0.0.0.0 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for CPU-only workstation (Ryzen 9950X, 192 GB DDR5)

Docker and enough system RAM for the model plus runtime overhead.
Substantial free disk space for the model cache.

llama.cpp (CPU Docker image) backend documentation ↗

Why this recipe

CPU inference is portable but usually much slower than a supported GPU backend.
`-t -1` lets llama.cpp choose the thread count. Set an explicit value if you need to leave CPU capacity for other services.
Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.
Q4_0 is the exact downloadable artifact. ModelRigs uses its closest catalogue profile, Q4_K_M, for the memory estimate and only shows comfortable fits to preserve headroom.

Artifact and assumptions

Repository

ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF Estimated

Pinned revision

8a08a1c81dad Estimated source

Artifact quantization

Q4_0 Estimated

Memory model ⓘ

Q4_K_M (nearest catalogue profile) Assumption

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

CPU only Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.