How to run Mistral Small 4 119B-A6.5B locally

Select your machine for accelerator-specific, revision-pinned commands using llama.cpp.

Choose your hardware

Showing 14 compatible configurations at 8K context with extra headroom for the UD-Q4_K_M artifact. Selection changes the instructions, not the canonical URL.

Recommended backend

Apple Metal

Calculated macOS

Artifact

UD-Q4_K_M

Estimated GGUF · unsloth/Mistral-Small-4-119B-2603-GGUF

Memory required

72.7 GB

Calculated of 435.2 GB usable

Fit at 8K context

Comfortable

~91.2–131 t/s decode

Verification status: artifact and download instructions checked against the runtime repository on 2026-09-15; command generated from a unit-checked backend template; not executed on physical Mac Studio M3 Ultra 512 GB hardware.

Community conversion: this runnable GGUF is published by unsloth, not by the model publisher. Its exact repository and immutable revision are shown below.

Text-only quick start: this recipe starts a text chat. Using the model's vision capability can require a matching multimodal projector and additional llama.cpp flags.

Quick start · llama.cpp (native)Not hardware tested

1. Install llama.cpp (native)

brew install llama.cpp

2. Start a local chat

llama-cli -hf "unsloth/Mistral-Small-4-119B-2603-GGUF:UD-Q4_K_M" -c 8192 -ngl 99 -cnv

The first run downloads the model into a persistent cache. Large artifacts can take a long time; make sure the machine has enough free disk space before starting.

Run an OpenAI-compatible local API

Start the server

llama-server -hf "unsloth/Mistral-Small-4-119B-2603-GGUF:UD-Q4_K_M" -c 8192 -ngl 99 --host 127.0.0.1 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for Mac Studio M3 Ultra 512 GB

A current macOS release on Apple Silicon.
Homebrew, or a current llama.cpp binary installed another way.
Enough free unified memory for the weights, KV cache and macOS.

llama.cpp (native) backend documentation ↗

Why this recipe

Metal support is enabled automatically in standard macOS llama.cpp builds.
Run llama.cpp natively: Docker Desktop cannot pass the Apple GPU through to a Linux container.
Use the chat template embedded in the GGUF; a hand-written prompt format can produce malformed or lower-quality responses.
For split GGUFs, llama.cpp downloads the remaining shards automatically. Allow plenty of disk space and time on the first run.
UD-Q4_K_M is the exact downloadable artifact. ModelRigs uses its closest catalogue profile, Q4_K_M, for the memory estimate and only shows comfortable fits to preserve headroom.

Artifact and assumptions

Repository

unsloth/Mistral-Small-4-119B-2603-GGUF Estimated

Pinned revision

bd93c721735a Estimated source

Artifact quantization

UD-Q4_K_M Estimated

Memory model ⓘ

Q4_K_M (nearest catalogue profile) Assumption

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.