How to run Granite 4.2 30B locally

Select your machine for accelerator-specific, revision-pinned commands using llama.cpp.

Model benchmarks

Choose your hardware

Machine

Showing 26 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Recommended backend

AMD ROCm

Calculated Linux

Artifact

Q4_K_M

Official spec GGUF · ibm-granite/granite-4.2-30b-GGUF

Memory required

20.5 GB

Calculated of 105.0 GB usable

Fit at 8K context

Comfortable

~5.6–8.1 t/s decode

Verification status: artifact and download instructions checked against the runtime repository on 2026-09-15; command generated from a unit-checked backend template; not executed on physical Framework Desktop (Ryzen AI Max+ 395, 128 GB) hardware.

Quick start · llama.cpp (ROCm Docker image)Not hardware tested

1. Start a local chat

docker run --rm -it \
  --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:full-rocm -hf "ibm-granite/granite-4.2-30b-GGUF:Q4_K_M" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

docker run --rm \
  --device=/dev/kfd --device=/dev/dri --group-add=video --ipc=host \
  -p 127.0.0.1:8080:8080 \
  -v llama-model-cache:/root/.cache/huggingface \
  ghcr.io/ggml-org/llama.cpp:server-rocm -hf "ibm-granite/granite-4.2-30b-GGUF:Q4_K_M" -c 8192 -ngl 99 --host 0.0.0.0 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for Framework Desktop (Ryzen AI Max+ 395, 128 GB)

A Linux distribution and kernel supported by the installed AMD driver/ROCm release.
Docker with `/dev/kfd` and `/dev/dri` available on the host.
Your exact GPU architecture must be supported by both ROCm and the published llama.cpp image.

llama.cpp (ROCm Docker image) backend documentation ↗

Why this recipe

ROCm compatibility is more hardware- and OS-specific than CUDA. Confirm the exact GPU and Linux release in AMD's compatibility matrix before downloading a large model.
The ROCm image and AMD device passthrough are intentional; a generic llama.cpp image is not sufficient for GPU acceleration.
Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.

Artifact and assumptions

Repository

ibm-granite/granite-4.2-30b-GGUF Official spec

Pinned revision

27b350a791e8 Official spec source

Artifact quantization

Q4_K_M Official spec

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.