How to run Ternary Bonsai 2 27B locally

Select your machine for accelerator-specific, revision-pinned commands using llama.cpp.

Model benchmarks

Choose your hardware

Showing 23 compatible configurations at 8K context. Selection changes the instructions, not the canonical URL.

Recommended backend

NVIDIA CUDA

Calculated Linux

Artifact

PTQ1_0

Official spec GGUF · prism-ml/Ternary-Bonsai-2-27B-gguf

Memory required

7.2 GB

Calculated of 22.1 GB usable

Fit at 8K context

Comfortable

91.1 t/s decode

Verification status: artifact and download instructions checked against the runtime repository on 2026-09-18; command generated from a unit-checked backend template; not executed on physical Used RTX 4090 workstation (24 GB) hardware.

Text-only quick start: this recipe starts a text chat. Using the model's vision capability can require a matching multimodal projector and additional llama.cpp flags.

Quick start · Prism ML llama.cpp forkNot hardware tested

1. Install Prism ML llama.cpp fork

git clone https://github.com/PrismML-Eng/llama.cpp.git
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build -j
hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PTQ1_0.gguf --revision 6ed5e12bf84b7a63069882c91dd9e9218647d17b --local-dir .

2. Start a local chat

./build/bin/llama-cli -m ./Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 8192 -p "Explain local AI hardware in one paragraph." -n 256

The install step downloads the pinned model file into the runtime directory. Keep that file alongside the chat and server commands.

Run an OpenAI-compatible local API

Start the server

./build/bin/llama-server -m ./Ternary-Bonsai-2-27B-PTQ1_0.gguf -ngl 99 -fa on -c 8192 --host 127.0.0.1 --port 8080

Test it from another terminal

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Reply with one short sentence: what hardware are you running on?"}]}'

The server binds to loopback by default. Do not expose the server to your network or the public internet until you have added authentication and appropriate access controls.

Prerequisites for Used RTX 4090 workstation (24 GB)

Git, CMake, a C++ build toolchain and the Hugging Face CLI.
A working NVIDIA driver and CUDA toolkit.

Prism ML llama.cpp fork backend documentation ↗

Why this recipe

The pinned model file is downloaded during installation; use the same directory for the chat and server commands.
The PTQ1_0 file needs Prism ML's Hadamard-aware llama.cpp fork. Stock llama.cpp cannot run this packing.
The 5.95 GB file contains the language model only. Image input also needs the optional mmproj file and additional memory.
The commands use the publisher's model artifact and runtime fork, but have not been run on the listed hardware.

Artifact and assumptions

Repository

prism-ml/Ternary-Bonsai-2-27B-gguf Official spec

Pinned revision

6ed5e12bf84b Official spec source

Artifact quantization

PTQ1_0 Official spec

Context in commands ⓘ

8,192 tokens Assumption

GPU layers ⓘ

99 (request full offload) Assumption

Hardware execution

Not tested by ModelRigs Estimated

Operational notes

Memory and context

Start at 8K. Longer prompts grow the KV cache; return to the fit calculation before raising the context substantially.

Persistent downloads

Docker recipes store Hugging Face downloads in the named volume llama-model-cache. Removing that volume also removes the cached weights.

Performance

The page reports measured throughput when available and otherwise shows an estimate range. Runtime build, drivers, prompt length and thermals can all move it.