modelrigs

Mistral Small 3.2 24B on Framework Desktop (Ryzen AI Max+ 395, 128 GB)

24.011B on 128 GB at 256 GB/s. Hardware details · Model details

Run locally →
Yes. Mistral Small 3.2 24B fits on Framework Desktop (Ryzen AI Max+ 395, 128 GB) at the recommended Q8_0 configuration, requiring approximately 27.2 GB at 8K context. Practical context capacity is 128K. Expected decode for the recommended configuration is ~3.9–5.7 t/s. Estimated
CompatibilityComfortable
Calculated fit
Yes
Calculated Uses under 80% of usable memory. Room for a long context and other work.
Recommended quantization
Q8_0
Calculated gguf
Memory required
27.2 GB
Calculated of 105.0 GB usable — 26%
Max practical context
128K
Calculated model supports 128K

Memory budget at 8K context

Model weights ⓘ
24.2 GB Calculated
KV cache ⓘ
1.3 GB Calculated
Runtime overhead ⓘ
1.7 GB Estimated
Total required
27.2 GB Calculated
Headroom ⓘ
77.7 GB Calculated
Highest-precision quantization that leaves headroom: uses 26% of usable memory at 8K context.
Unified LPDDR: the accelerator allocates from the same 128 GB pool as the OS. We assume 82% is available to the inference engine.
Performance

Estimated performance · recommended Q8_0

Decode, prefill and TTFT below are estimates for Q8_0. We do not have a comparable Q8_0 measurement on this machine.

Estimated decode
~3.9–5.7 t/s
Estimated Q8_0; calculated range
Estimated prefill
~144–299 t/s
Estimated Q8_0; calculated range
Estimated TTFT at 8K
~27.5–57.0 s
Estimated Q8_0; calculated range
Hardware load reference
130 W
Measured machine-level load; not this model run
How the estimate is calculated
  1. Decode: reading 24.0B active parameters at 8.50 bits/weight takes 206.62 ms at 256 GB/s x 48% achieved efficiency.
  2. Prefill: 59 TFLOPS (FP16) x 18% assumed model-FLOPs utilisation, divided by 2 x 24.0B parameters per token.
  • Prefill throughput is highly engine-dependent. Flash attention, batch size and quantized KV all move this number substantially, and for sparse mixture-of-experts models it is the least reliable figure we produce.
  • This is a calculated estimate, not a measurement. It assumes a single request, a short prompt, no speculative decoding and a warm model already resident in memory.

Estimator version estimator_v1. Stored with every estimated row so old estimates can be regenerated when the model improves.

Submit a benchmarkSubmissions stay pending until reviewed.
This model at other quantizations on this machine
QuantizationResidentDiskTotal neededUtilisationFitMax context
Q4_K_M14.0 GB14.0 GB16.6 GB16%Comfortable128K
Q5_K_M16.4 GB16.4 GB19.1 GB18%Comfortable128K
Q8_0recommended24.2 GB24.2 GB27.2 GB26%Comfortable128K
BF1644.7 GB44.7 GB48.3 GB46%Comfortable128K
Cost of running Mistral Small 3.2 24B on this machineUnited States (federal) · C corporation · 8h/day
Monthly economic cost
$28
Calculated after tax
Codex/Claude Code
$100/mo
≈ $100/mo · local is $72 less
Net cash at purchase
$2,161
Calculated VAT not reclaimable
Total over 5 years
$1,673
Calculated after tax, after resale
Cost per USD/1M tokens
$45.80
Calculated 0.6M tokens/month

The $100 comparison uses the Codex Pro 5x / Claude Max 5x plans. A fixed planning conversion is used for USD. This compares monthly spend only: subscriptions have usage limits, local hardware has different capabilities and constraints, and taxes or regional pricing may change the charged amount. Prices checked 7 September 2026.

Purchase-price input: $2,161 in United States (federal), tax/VAT excluded · estimated · Framework (converted catalogue estimate). This is the localized purchase input; the after-tax economic cost is calculated separately below.

Monthly breakdown

Depreciation
1,837 over 5 years, straight-line to a 324 residual
$30.62
Electricity
6.3 kWh/month at 0.140/kWh
$0.88
Cost of capital
4.0%/yr on 1,243 average capital employed
$4.14
Monthly cost before tax$35.63
Electricity tax shield
Running costs are deductible business expenses
−$0.18
First-year expensing
§179 (100.0%)
−$7.56
Monthly economic cost after tax$27.89

Three different numbers, deliberately

Cash cost
$2,161
Money that leaves the bank account on day one, net of reclaimable VAT.
Accounting depreciation
$30.62/month
$1,837 written down over 5 years to a $324 residual.
After-tax economic cost
$27.89/month
Depreciation plus running costs plus cost of capital, less the tax those deductions save. This is the figure to compare between machines.
Assumptions and sources (verified 2026-09-07)
VAT rate
0% Official spec
VAT recoverable
0% Official spec
Federal C-corporation estimate at the flat 21% rate. Pass-through entities should use the sole-proprietor estimate as a rougher proxy.
Effective deduction rate
21.00% Calculated
Headline marginal rate 21.00%.
Depreciation
5 years, straight-line Assumption
Residual value
$324 (15%) Assumption
Two GPU generations later, the card is worth a fraction of its list price.
Electricity
$0.140/kWh Assumption
Average power draw
36 W Calculated
Load 130 W for 20% of powered hours, idle 12 W for the rest — a machine that is on is not generating tokens the whole time.
Investment allowances
§179 100.0% Official spec
Worth $454 in total — first-year expensing that replaces later tax depreciation.
Cost of capital
$4.14/month Assumption
Assumes the machine generates tokens 20% of its 176 powered hours per month. Cost per token scales inversely with this number — halve the utilisation and the cost per token doubles.
Federal planning estimate only, not tax advice. State and local income tax, sales/use tax and incentives are excluded. It assumes 100% business use, a Section 179 election, enough business income to use it, and that the full annual limit remains available.
Efficiency metrics
Decode per $1,000 spent
2.2 t/s
Calculated purchase price only
Decode per $100/month
17.2 t/s
Calculated after-tax ownership cost
Tokens per joule
0.13
Calculated same as tokens/s per watt
USD per 1M output tokens
$45.80
Calculated at 20% utilisation
Local versus hosted APIs
Hosted modelUSD/1M outputBreak-evenAPI at your volumeVerdictComparison type
Qwen3.8 Flash
Alibaba Cloud
$0.4218.1M/mo$1API cheaperDifferent model
DeepSeek-V4.1 Flash
DeepSeek
$0.6015.5M/mo$1API cheaperDifferent model
DeepSeek-V4 Pro
DeepSeek
$1.983.8M/mo$4API cheaperDifferent model
DeepSeek V4 Pro (Together)
Together AI
$4.401.3M/mo$13API cheaperDifferent model
GLM-5.3
Z.ai
$4.401.8M/mo$9API cheaperDifferent model
Claude Haiku 4.5
Anthropic
$5.002.1M/mo$8API cheaperDifferent model
Different-model comparison: the hosted model is not the model you would run locally. Treat this as a workload-quality trade-off, not a direct economic equivalence — the frontier model may complete a task in fewer tokens, or complete tasks the local model cannot. Break-even is the monthly output volume at which API spend equals the $28/month economic cost of owning this machine, assuming 8 input tokens per output token. Hosted prices are published in USD. This table is the canonical US default.
Mistral Small 3.2 24B on other hardware
HardwareDecodePriceFit
NVIDIA DGX H200 (8x H200, 1,128 GB)~351–506 t/s$359,430 *Comfortable
Quad RTX 5090 workstation (4x 32 GB)~85.7–123 t/s$12,611 *Comfortable
Lenovo ThinkStation PX (4x RTX PRO 6000, 384 GB)~75.1–108 t/s$58,553 *Comfortable
RTX 5090 workstation (1x 32 GB)~63.9–91.9 t/s$3,332 *Comfortable
Dual RTX 5090 workstation (2x 32 GB)~57.6–82.8 t/s$5,854 *Comfortable
Dell Precision 7960 Rack (2x RTX PRO 6000, 192 GB)~52.8–76 t/s$36,663 *Comfortable
Nearest alternatives to the Framework Desktop (Ryzen AI Max+ 395, 128 GB)
Mistral Small 3.2 24B on GMKtec EVO-X2 (Ryzen AI Max+ 395, 128 GB)128 GBMistral Small 3.2 24B on Framework Desktop (Ryzen AI Max+ 395, 64 GB)64 GBMistral Small 3.2 24B on NVIDIA DGX Spark 128 GB128 GBMistral Small 3.2 24B on Radeon AI PRO R9700 workstation (32 GB)32 GBMistral Small 3.2 24B on Mac Studio M5 Max 128 GB128 GB
Compare Framework Desktop (Ryzen AI Max+ 395, 128 GB) against GMKtec EVO-X2 (Ryzen AI Max+ 395, 128 GB) →