modelrigs

How to run Nemotron 3.5 Lightning 30B-A3B locally

Select your machine for accelerator-specific, revision-pinned commands using llama.cpp.

Model benchmarks
Choose your hardware

Showing 24 compatible configurations at 8K context with extra headroom for the Q4_0 artifact. Selection changes the instructions, not the canonical URL.

Recommended backend
Apple Metal
Calculated macOS
Artifact
Q4_0
Estimated GGUF · ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF
Memory required
20.0 GB
Calculated of 30.6 GB usable
Fit at 8K context
Comfortable
~165–237 t/s decode
Verification status: artifact and download instructions checked against the runtime repository on 2026-09-15; command generated from a unit-checked backend template; not executed on physical Mac Studio M5 Max 36 GB hardware.
Quick start · llama.cpp (native)Not hardware tested
1. Install llama.cpp (native)
brew install llama.cpp
2. Start a local chat
llama-cli -hf "ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API
Start the server
llama-server -hf "ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_0" -c 8192 -ngl 99 --host 127.0.0.1 --port 8080
Test it from another terminal
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'
The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.
Prerequisites for Mac Studio M5 Max 36 GB
  • A current macOS release on Apple Silicon.
  • Homebrew, or a current llama.cpp binary installed another way.
  • Enough free unified memory for the weights, KV cache and macOS.

llama.cpp (native) backend documentation ↗

Why this recipe
  • Metal support is enabled automatically in standard macOS llama.cpp builds.
  • Run llama.cpp natively: Docker Desktop cannot pass the Apple GPU through to a Linux container.
  • Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
  • For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.
  • Q4_0 is the exact downloadable artifact. ModelRigs uses its closest catalogue profile, Q4_K_M, for the memory estimate and only shows comfortable fits to preserve headroom.
Artifact and assumptions
Repository
ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF Estimated
Pinned revision
8a08a1c81dad Estimated source
Artifact quantization
Q4_0 Estimated
Memory model ⓘ
Q4_K_M (nearest catalogue profile) Assumption
Context in commands ⓘ
8,192 tokens Assumption
GPU layers ⓘ
99 (request full offload) Assumption
Hardware execution
Not tested by ModelRigs Estimated
Operational notes
Memory and context
Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.
Persistent downloads
Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.
Performance
The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.