How to run Qwen3.5 122B-A10B locally

Select your machine for an accelerator-specific command. This first iteration uses llama.cpp and a revision-pinned GGUF artifact.

Model benchmarks

Choose your hardware

Showing 17 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Recommended backend

NVIDIA CUDA

Calculated Linux

Artifact

Q4_K_M

Estimated GGUF · unsloth/Qwen3.5-122B-A10B-GGUF

Memory required

76.7 GB

Calculated of 88.3 GB usable

Fit at 8K context

Fits

~133–192 t/s decode

Verification status: artifact existence checked against Hugging Face on 2026-09-11; command generated from a unit-checked backend template; not executed on physical RTX PRO 6000 Blackwell workstation (96 GB) hardware.

Community conversion: this runnable GGUF is published by unsloth, not by the model publisher. Its exact repository and immutable revision are shown below.

Quick start · llama.cpp (CUDA Docker image)Not hardware tested

1. Start a local chat

docker run --rm -it \
  --gpus all \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full-cuda -hf "unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

docker run --rm \
  --gpus all \
  -p 127.0.0.1:8080:8080 \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:server-cuda -hf "unsloth/Qwen3.5-122B-A10B-GGUF:Q4_K_M" -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The Docker commands publish port 8080 on 127.0.0.1 only. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for RTX PRO 6000 Blackwell workstation (96 GB)

A working NVIDIA driver on the host.
Docker with NVIDIA Container Toolkit configured; `docker run --rm --gpus all nvidia/cuda:12.8.1-base-ubuntu24.04 nvidia-smi` should see the GPU.
Enough aggregate VRAM for the selected model. Multi-GPU machines also depend on llama.cpp's split and interconnect behaviour.

llama.cpp backend documentation ↗

Why this recipe

The `--gpus all` flag exposes NVIDIA GPUs to the container; the generic llama.cpp image would otherwise run on the CPU.
The named Docker volume keeps the downloaded model between runs.
Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.

Artifact and assumptions

Repository

unsloth/Qwen3.5-122B-A10B-GGUF Estimated

Pinned revision

51eab4d59d53 Estimated source

Artifact quantization

Q4_K_M Estimated

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.