skip to content
← Research

Conifer Bench · Memory

Fitting a 24B model to a 36 GB Mac without thrashing

Why the KV cache, not the weights, is what breaks a large model on consumer memory, and the memory-aware context sizing that keeps it fast.

Conifer BenchJune 202610 min read

A 24B model can be correct and unusable at the same time. Load Mistral Small 24B on a 36 GB Mac at its full native window and decode falls to roughly two thirds of a token per second, about a twenty-eighth of its healthy rate, while the answer it eventually produces is exactly right. Nothing is broken. The model asked for more memory than the machine has, and the operating system spends every token paying for it. The fix is to size the context window to the memory you have, not to the window the weights advertise.

0.65 tok/s24B, full native windowthe regression the engine reports
18.4 tok/s24B, fitted window~28× faster, corroborated by bench
21.5 GBKV cache at the native windowon top of 14.3 GB of weights
13,107tokens the 2 GiB cap allowsvs 131,072 native

A model that is correct and unusable at once

On a unified-memory Mac the CPU and GPU share one pool of RAM, and the KV cache for a run is pre-allocated up front. At load time the runtime reserves context_length × kv_bytes_per_token of anonymous memory and never grows it. Decode stays allocation-free, which is what you want, but it means the window you ask for is a memory bill you pay immediately, before the first token.

Mistral Small 24B ships at a native window of 131,072 tokens.1 Each token of KV for this geometry costs 163,840 bytes, exactly 160 KiB.2 Multiply the two and the cache alone wants 21.5 GB (20.0 GiB).3 The Q4_K_M weights are another 14.3 GB.4 Together that is 35.8 GB of working set against a 36 GB machine.5

It does not fit. The raw total sitting a few hundred megabytes under 36 GB is the trap, not the reprieve. Usable RAM after the OS, the app, the WebView, and decode scratch is well below 36 GB, so the working set crosses the real ceiling, the memory compressor evicts hot weight pages, and every decode step re-faults the pages it just lost. The 0.65 tok/s comes from there: not slow math, a machine swapping the model against itself.6 Measured on the bench, the same 24B runs at 18.56 tok/s at 512 tokens of context.7

24B working set on a 36 GB Mac          0        10       20       30  GB
                                        +--------+--------+--------+----
native window (131,072 tok)
  weights  14.3 GB  [##############                          ]
  KV       21.5 GB  [              #####################      ]
  total    35.8 GB  >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>  over usable RAM -> thrash

2 GiB-fitted window (13,107 tok)
  weights  14.3 GB  [##############                          ]
  KV        2.0 GB  [              ##                         ]
  total    16.3 GB  fits, decode runs at speed
Figure 1Working set at the native window vs. at the 2 GiB-fitted window. The KV cache, not the weights, is what tips the 24B over a 36 GB machine. 3

Fit the window to the memory, not the model

The default that springs the trap is requesting the model’s native ctx_len whether or not the run will ever use it. Invert that: leave the window unset and let the runtime back-solve it from a memory budget. When a run does not name a context length, the engine budgets a fixed 2 GiB of KV and divides, window = 2 GiB / kv_bytes_per_token.8 For the 24B that is 13,107 tokens, not 131,072.9 The cache it reserves drops from 21.5 GB to 2 GiB, and the working set falls back under the ceiling.

Capacity is a pure function of the model config and the requested length, recomputed every run and never stored.8 An explicit request still opts in to whatever you ask for, clamped to native: name a window and you get it. Only the un-named path takes the cap, and the cap is floored so the smallest models are never starved. DEFAULT_CTX_CAP is 8,192 tokens, the floor below which short prompts always work even when the byte budget computes something smaller.10 A model with very fat KV, 524,288 bytes per token in the engine’s test geometry, back-solves to a raw cap of 4,096 and is lifted to that floor.11

Counting KV bytes per token, honestly

The whole mechanism rests on one number being right: how many bytes of cache a single token costs. For each attention layer the cache stores keys and values, so 2 × n_kv_heads × head_dim elements per token, summed across layers, doubled for fp16.12 The subtlety is what to exclude. A recurrent or hybrid layer, the convolutional state in an lfm2 model or the DeltaNet state in a qwen3.5 SSM, carries a fixed state cache rather than a growing K and V, so it contributes zero per-token KV.12 The estimate mirrors the real allocator’s layer_elems exactly, so the number that sizes the window and the number actually reserved are the same. The budget can never quietly under-count.

For the 24B the arithmetic is 40 layers, 8 KV heads, head dimension 128, times 2 for K and V, times 2 for fp16, which is 163,840 bytes.2 Three witnesses agree on that figure: the engine’s own fixture, the arithmetic, and a hardcoded website test constant of 160 × 1024.13 It matches the published Mistral-Small-24B config.

KV bytes per token and the window the 2 GiB cap allows
Model / geometryB/tokenNative ctxCapped window
Mistral Small 24B (40L, 8 kv, hd 128)163,840131,07213,107
Fat-KV test geom. (64L, 8 kv, hd 256)524,288n/a8,192 (floor)
lfm2-8b-a1b (hybrid, ~recurrent)~10,240128,000128,000 (full)

Cap = 2 GiB / B-per-token, floored at DEFAULT_CTX_CAP = 8,192.9 The lfm2 per-token figure is an approximate comment value, not a measured constant.14

The cap discriminates by architecture

A blanket cap would punish the models that do not need it. This one does not, because the per-token cost drives it instead of a flat number. A hybrid model carries almost no KV, so even with little free RAM its full window fits the budget and it keeps native context. The website sizer makes this concrete: an lfm2-8b-a1b at roughly 10 KB per token, asked for its 128,000-token native window with only 6 GiB free, keeps the full 128,000.15 A big-KV dense transformer in the same conditions gets fitted down. Same rule, opposite outcome, because the two architectures cost different amounts to remember a token.

Under real pressure the floor takes over from the budget. With 1 GiB free, the budget after headroom goes negative and the window floors at 8,192 instead of computing something unusable.16 You get a window that always works and usually fits, with no knob to set.

window (tokens, log-ish)   free RAM ->   1 GiB    6 GiB    10 GiB
                                       +--------+--------+--------
24B   (160 KiB/tok)                    | 8,192  | ~19k   | ~46k    < native 131,072
hybrid (~10 KB/tok)                    | 8,192* | 128,000| 128,000   = native (full)

  *under 1 GiB free both floor at 8,192; the 24B stays floored where
   the hybrid would have kept its full window with any real headroom.
   ~46k for the 24B at 10 GiB free is computed from the sizer, not a
   shipped default.
Figure 2Window the website sizer picks as free RAM falls, for the 160 KiB/token 24B vs. a ~10 KB/token hybrid. The dense model is fitted down; the hybrid keeps native.15

Two halves of the mechanism, and the bug between them

The guard lives in two places, and the v1.2.4 fix closed the seam between them. First half, the engine: an un-named run takes the 2 GiB effective_capacity cap above. Second half, the desktop. At model load the adapter probes live free memory, reads kv_bytes_per_token off the engine’s ModelInfo struct, and computes floor((free - 3 GiB headroom) / kv_bytes_per_token), clamped to native and floored at 8,192.17 The 3 GiB it holds back covers decode scratch, the app, the WebView, and the OS, the slack that keeps 35.8 from passing for “fits.”

Now the bug the release fixed. The engine already capped un-named runs, but the desktop studio path always sent an explicit full-native window, which counts as a request and defeats the engine’s own guard. The model dutifully reserved 131,072 tokens of KV and thrashed. The fix omits context_length entirely on macOS and CPU, where there is no separate VRAM ceiling to respect, so the engine’s memory-aware cap applies as designed.18 The user-facing default for Max context became 0, meaning “use the native window,” and the old broken default of 4,096, which had silently capped every chat, migrates to 0.19 Sage v1.2.4 carries all of this.20

What was measured, and what stands guard

The recovery that was actually benched end-to-end is on a 7B, not the 24B. Qwen2.5-7B at the same 131,072 native window came down to 0.80 GiB resident at 51.8 tok/s under the cap, against roughly 16 GiB uncapped.22 A clean before-and-after: same fix, a twentyfold drop in resident memory, full speed retained. For the 24B the evidence is the two ends of the line, not the transition itself. The thrash figure of 0.65 tok/s is what the engine reports for the uncapped run.6 The healthy curve under a fitted window holds between 18.56 and 16.75 tok/s from 512 to 8,192 tokens.7 Over the same depths llama.cpp on the identical model sags from 18.14 to about 12 to 13 tok/s, the shape of a decode path under depth pressure.23

Prevention is the window fit. The backstop sits separate and lower in the stack: when the cache actually allocates, it estimates total fp16 KV bytes and hard-fails with a KvOom error if the request exceeds the device budget the runtime passed.24 A window that should never have been requested is refused outright rather than allowed to swap. The sizing keeps you off that path; the OOM guard catches the case the sizing did not.

Measured 24B decode under a fitted window, vs. llama.cpp at depth
ContextConifer tok/sllama.cpp tok/s
51218.5618.14
102418.47n/a
204818.49n/a
409618.0612.35
819216.7513.02

Mistral Small 24B Q4_K_M, M3 Max 36 GB.7 The llama.cpp column is a different engine, shown only as a depth-decode contrast, not as the thrash bug.23 The 24B was not benched end-to-end at its fitted ~13K window to capture the thrash-to-recovery transition; the verified recovery is the 7B.22

The change is small and the result is unglamorous: a model that was already correct now also runs. The lesson generalizes past this one model. On a machine where compute and memory share a pool, the context window is not a quality setting, it is a memory allocation, and the right default sizes it to the RAM in front of you, not the number printed on the weights.

References & method

  1. 1Native context length 131,072 for Mistral Small 24B, taken from the model’s published config and pinned in the Conifer catalog. The same 131,072 is asserted as a fixed constant in the windowing unit tests, so the figure the article cites and the figure the sizer uses are the same.
  2. 2Per-token KV cost 163,840 B (160 KiB) for the 24B geometry (40 layers, 8 KV heads, head_dim 128). Derived from the model geometry rather than parsed from a live GGUF, and cross-checked against the published Mistral-Small-24B config. The number is triply corroborated: the engine’s cap test fixture, the closed-form arithmetic (40 × 2 × 8 × 128 × 2), and an independent hardcoded website constant all land on 163,840.
  3. 3Full-window KV = 163,840 × 131,072 = 21,474,836,480 B = 20.0 GiB = 21.5 GB. Computed from the per-token figure (note 2) and native ctx (note 1); the engine’s own sizing notes round this to “~21 GB.” Binary and decimal are the same figure.
  4. 4Q4_K_M weight size 14,333,910,496 B = 14.33 GB for Mistral Small 24B, read as the exact byte size of the shipping quantized weight file in the Conifer catalog. The catalog ships the 3.1 (2503) build; the bench tok/s rows (note 7) are the 3.2 (2506) build, same geometry.
  5. 5Total working set = 14,333,910,496 + 21,474,836,480 = 35,808,746,976 B = 35.81 GB = 33.35 GiB. Computed. Host is Apple M3 Max, 14-core CPU / 30-core GPU, 36 GB unified memory, the machine recorded in the benchmark run header.
  6. 6The 0.65 tok/s thrash rate is the regression the Conifer engine records in its own memory-aware KV cap (“the measured mistral-24b 0.65 vs 18.4 tok/s regression”), the before-figure that motivated the cap. It is a single reported measurement, not a figure reproduced in the open benchmark suite, so treat it as the regression the engine reports rather than a re-derivable number. That 35.81 GB sits under 36 GB in raw bytes does not mean it fits: usable RAM after OS, app, WebView, and decode scratch is well below 36 GB.
  7. 7Healthy 24B decode curve, Conifer engine, best-of-rep per depth, for Mistral Small 3.2 24B Instruct 2506 Q4_K_M: 18.56 @512, 18.47 @1024, 18.49 @2048, 18.06 @4096, 16.75 @8192 tok/s. Token-generation throughput swept across context depths on an Apple M3 Max with 36 GB unified memory, from the general decode benchmark suite. This is the post-cap healthy regime, not a measurement of the thrash-to-recovery transition.
  8. 8The engine’s effective_capacity rule: an explicit request clamps to native (Some(x) → x.min(ctx_len).max(1)); an un-named run takes ctx_len.clamp(1, (2 GiB / per_tok).max(8192)) against a fixed 2 GiB KV pre-allocation budget. A pure function of (config, requested), recomputed each run and never stored, so it can be reasoned about directly from the model config.
  9. 9Capped window 13,107 tokens for the 24B, the value 2 GiB / 163,840 B back-solves to and the exact figure pinned by the engine’s windowing unit test (effective_capacity(mistral, None) == 13_107).
  10. 10DEFAULT_CTX_CAP = 8192, the engine’s window floor and the predecessor of the byte-aware budget. The change history records the byte budget as a port of the earlier resolve_kv_capacity logic into the current config-driven engine, so it is a refinement of an existing cap, not a new invention.
  11. 11Sub-floor case, exercised by an engine unit test: a fat-KV geometry of 64 layers, 8 KV heads, head_dim 256 = 524,288 B/token back-solves to a raw cap of 4,096 and is lifted to DEFAULT_CTX_CAP = 8,192.
  12. 12The engine’s KV-bytes formula: elems += 2 × n_kv × head_dim per attention layer, total × 2 for fp16; recurrent/hybrid layers (lfm2 Conv, qwen3.5 DeltaNet) contribute 0 because they hold state caches, not K/V. The estimator shares the same per-layer element count as the real cache allocator, so the number that sizes the window and the number actually reserved agree by construction. All math assumes fp16 KV; with quantized KV the real allocation is smaller than this estimate.
  13. 13Second independent witness to 163,840 B/token: a hardcoded constant, MISTRAL_KV = 160 * 1024, baked into the desktop windowing tests separately from the engine geometry.
  14. 14The lfm2 ~10 KB/token figure is an approximate illustrative value (the engine notes ~12 KB, the desktop windowing test uses 10 KB), not a measured per-model constant. Reported as “roughly” for that reason.
  15. 15Asymmetry test of the desktop sizer: memoryAwareContextWindow(128000, 10 * 1024, 6 * GiB) returns 128,000, so a ~10 KB/token hybrid keeps its full native window with only 6 GiB free. The ~46k figure for the 24B at 10 GiB free is computed from the same sizer formula (floor((10 GiB - 3 GiB) / 160 KiB) = 45,875), not a shipped default.
  16. 16Heavy-pressure floor, exercised by the desktop sizer test: at 1 GiB free the budget after the 3 GiB headroom goes negative and the window floors at MEMORY_AWARE_CTX_FLOOR = 8,192.
  17. 17The desktop memoryAwareContextWindow sizer: floor((available - KV_AUTO_HEADROOM_BYTES) / kvBytesPerToken), clamped to native, floored at 8,192, with KV_AUTO_HEADROOM_BYTES a fixed 3 GiB. The per-token KV cost is surfaced from the engine to the desktop on the model-info struct returned at load (defaulted to 0 for an older persisted struct, which then falls back to native). The window is sized once at model load from a live free-memory probe; a null probe falls back to native.
  18. 18Root cause and fix, in the desktop studio request adapter: the desktop always sent an explicit full-native window, which counts as a request and defeats the engine guard. The fix omits context_length on macOS and CPU (no separate VRAM ceiling to respect) so the engine applies its own un-named-run effective_capacity cap.
  19. 19The user-facing Max context default became 0 (“full native window”), and the old broken default of 4,096, which had silently capped every chat, is migrated to 0 on load (if (merged.maxContext === 4096) merged.maxContext = 0).
  20. 20Shipped in Sage v1.2.4, the version pinned in the desktop app’s build config ("version": "1.2.4").
  21. 21Two constants on two code paths: the engine KV_PREALLOC_BUDGET (a fixed 2 GiB) on the un-named-run auto-cap path, and the desktop KV_AUTO_HEADROOM_BYTES (3 GiB) on the live-memory sizer path. On macOS studio chat the desktop omits the context length and the engine cap is the operative guard. The 2 GiB budget is a fixed constant, not scaled to host free RAM.
  22. 22Bench-verified resident-memory recovery on qwen2.5-7b (native ctx 131,072): 0.80 GiB resident at 51.8 tok/s under the cap vs. ~16 GiB uncapped, an internal before-and-after run of the same fix recorded in the engine evidence pack. The 24B was not benched end-to-end post-fix at its fitted ~13K window.
  23. 23llama.cpp depth-decode contrast on the identical 24B model: 18.14 @512, 12.35 @4096, 13.02 @8192 tok/s, token-generation throughput from the same general decode benchmark suite on the same M3 Max 36 GB host. A different engine, shown only as a depth-decode contrast, not the thrash bug.
  24. 24OOM backstop in the engine’s cache allocator: at allocation time it estimates total fp16 KV bytes (total_elems * 2) and returns a KvOom error when the request exceeds the device budget the runtime passes. Prevention is the window fit; this is the backstop.