Where Conifer beats MLX on Apple silicon, and where MLX still wins

A new decode-attention kernel in the Conifer engine reads each KV tile once per query group instead of once per head. The payoff grows with context: at a 32k window it runs Llama-3.1-8B 2.9 times faster than our own previous kernel. That win is real, and it is internal. At true byte-parity on the same 8B model, MLX still finishes the token first at every depth we measured. Both facts hold, and the gap that remains is mostly the hardware.

Decode on a local machine is memory-bound. Every new token re-reads the weights and the growing KV cache from unified memory, so sustained tokens per second comes down to how many bytes the chip moves per step and how little of that traffic is wasted. Everything below runs on one Apple M3 Max, 14-core CPU and 30-core GPU, with 36 GB of unified memory.¹ All three engines, Conifer, MLX, and llama.cpp, run the same models on the same box.

The bug worth fixing was a redundant read#

Llama-3.1-8B uses grouped-query attention: 32 query heads share 8 key and value heads, a 4-to-1 ratio. The previous Conifer kernel split the work per query head, so each of the 4 query heads in a group re-read the same KV tile from memory independently. For a memory-bound kernel, reading the same bytes four times is the whole problem. That redundancy is fixed per token, but the KV cache it re-reads grows with the conversation, so the wasted fraction climbs the longer you talk.

The replacement is a cooperative-load GQA kernel. It loads each KV tile into threadgroup memory once, then emits the result for all four query heads in the group from that single read. None of this is a Conifer invention. Split-KV and flash-decoding live in both MLX and llama.cpp. The work here is the GQA cooperative load, templating the kernel across head dimensions {64, 128, 256, 512}, wiring it onto one reused partials reduction, and gating it per shape from measurement rather than intuition.⁴

Llama-3.1-8B decode  (tok/s)        new = coop kernel, prior = per-head
ctx     new     prior   ratio
  512   49.2 ┤████████████████████  50.0   0.98×  (slight regress)
 2048   48.1 ┤███████████████████   41.1   1.17×
 4096   45.2 ┤██████████████████    33.5   1.35×
 8192   39.8 ┤████████████████      25.7   1.55×
16384   31.7 ┤█████████████         16.2   1.96×
32768   22.9 ┤█████████              8.0   2.86×
        └────┴────┴────┴────┴────┴
        0    10   20   30   40   50  tok/s

ratio of new:prior climbs monotonically with context  (0.98× → 2.86×)

At 512 tokens the new kernel is a fraction slower, because at a short window the redundant read is cheap and the cooperative load carries a little setup cost.⁵ The two lines cross almost immediately, and from there the gap only widens. By 16k the new kernel is roughly twice as fast. By 32k it holds 23.0 tok/s where the old one has collapsed to 8.0. That is the shape you want from a fix that targets traffic: invisible when traffic is small, decisive when it dominates.

In isolation, the matvec reads more bandwidth than MLX’s#

One level down from the full attention path is the 4-bit matrix-vector kernel that decode spends most of its time in. To compare it against MLX fairly, the engine carries a faithful in-tree replica of MLX’s actual qmv_fast kernel structure and races it against Conifer’s production matvec on identical shapes, same cold-DRAM harness, median of nine.⁶ On the 8B feed-forward shape Conifer reads 3 to 8 percent more memory bandwidth. On a square 5120-by-5120 shape the margin is 13 to 17 percent, about 80 percent bandwidth utilization against MLX’s 67.

The clean, repeatable win is against llama.cpp#

The most reproducible result is not against MLX at all. At byte-parity on Llama-3.1-8B, pure-Q4K at roughly 4.5 bits per weight across all three engines, Conifer beats llama.cpp at decode and the lead widens sharply with depth.² Same redundant-read problem, this time in llama.cpp’s attention path: its decode falls off about 4-fold from 512 to 32k, while Conifer holds the drop to 2.3-fold.

Llama-3.1-8B decode, byte-parity (~4.5 bpw)   tok/s
ctx     MLX     Conifer   llama.cpp
  512   59.6    52.2      50.7
 2048   56.1    50.9      45.5
 8192   47.5    41.5      30.9
16384   34.5    29.0      19.9
32768   25.4    22.6      12.9

                 MLX ┤██████████████████████████  59.6 @512
             Conifer ┤███████████████████████     52.2
           llama.cpp ┤██████████████████████       50.7
                     │
                 MLX ┤███████████               25.4 @32768
             Conifer ┤██████████                22.6
           llama.cpp ┤█████                     12.9
                     └────┴────┴────┴────┴────┴────┴
                     0   10   20   30   40   50   60  tok/s

vs llama.cpp:  +3% @512 → +34% @8k → +75% @32k
vs MLX:        0.88× @512 … 0.89× @32k   (MLX ahead throughout)

Prefill tells the same two-baseline story. At depth Conifer pulls away from llama.cpp, plus 67 percent at 8k and plus 91 percent at 16k, while MLX stays ahead at every depth, 0.77 to 0.95 times its rate.⁷ On the owner’s real models the pattern repeats. Against llama.cpp: plus 57 percent on Gemma-3-4B, plus 90 percent on Qwen2.5-7B, plus 48 percent on Gemma-3-12B. Against MLX, those same mid-size models trail by 15 to 27 percent.⁸ Conifer is the engine that beats llama.cpp. MLX is the frontier still ahead of it.

Where Conifer does beat MLX end-to-end, and where it doesn’t#

End-to-end against MLX, the result is model-dependent. In the most recent byte-matched run, Conifer wins decode outright at every depth on three of five dense models, and loses on the other two.³ The wins are the tiny dense models and the mixture-of-experts model. The losses are the mid-size dense models.

The split is not random. Conifer wins the models where a token is cheap, so fixed per-token overhead dominates, and Conifer’s overhead is low: the token chain stays on the GPU and argmax runs in two passes rather than a host round-trip. On a 1B model finishing a token in well under five milliseconds, shaving that fixed cost is the whole race. The MoE wins for a related reason. It activates a small slice of its parameters per token, so it too is overhead-sensitive. None of this carries to a mid-size dense model, where the matvec work itself sets the pace, and there MLX’s leaner path still wins.

The last 10 percent is hardware, and the proof was a quiet box#

Conifer’s individual kernels read more bandwidth in isolation, yet MLX keeps its lead end-to-end. The decode-war census explains why. Running on a quiet box, gated so no other process used real CPU and GPU utilization sat below 20 percent before each run, decode on the M3 Max is already 95.5 to 95.7 percent GPU-busy, with only about 0.27 milliseconds per token of idle, and that idle is flat with depth.⁹ The GPU is saturated. No large recoverable idle is left to attack with smarter scheduling.

One number resolves the paradox. At a 2048 window MLX runs a whole token in roughly 4.7 to 4.9 milliseconds. Conifer’s GPU-active time alone, before any idle, is 5.76 milliseconds, longer than MLX’s entire token.¹⁰ Conifer’s individual GEMV kernels already beat MLX’s, so the lead cannot be faster kernels. It is less total GPU work per token: a leaner attention path with fewer dispatches. That difference is structural, not a tuning gap.

The remaining gap is a high-risk, single-digit-percent project, and the map of dead ends says so. Fused QKV bought 0.8 percent and was rejected. A norm-fusion attempt regressed 9.7 percent. Skipping the attention reduction is not sound under Metal’s memory model. The M3 matrix engine’s practical ceiling sits near 7 to 8 TFLOP/s, about 38 percent of the chip’s nominal figure, which is the wall these levers keep hitting.¹² The honest read: the last 10 to 20 percent against MLX is mostly the hardware and the attention path, not a kernel Conifer has failed to write.

How the numbers were taken#

Two ground rules keep the comparison fair. First, byte-parity: the MLX comparison uses pure-Q4K at roughly 4.5 bits per weight across all three engines, so no one wins on a lighter quantization. Where a run is not byte-matched, the text says so. Second, the host peaks are estimates, not measurements. The 300 GB/s bandwidth and 21.3 TFLOP/s figures come from a device registry, flagged as estimated. The empirically measured peak on this chip is 270.93 GB/s by a Metal triad microbenchmark, about 90 percent of spec.¹³ So any memory-bandwidth-utilization number framed against the 300 roofline understates true utilization.

Everything here runs on one M3 Max under thermal throttling, where absolute tokens per second can swing by about a factor of two between cold and warm runs. Only ratios and same-thermal-window interleaved A/B comparisons are trustworthy, which is why this note reports ratios and reads them best-of-rep. The cooperative-load and prior kernels are verified bit-identical to a CPU oracle across GQA group ratios of 4, 7, and 8.¹⁴ The newer flash-vec variant is online-softmax equivalent rather than bit-identical, and it is gated by a CPU-reference microbench and a greedy A/B rather than a dedicated case in the blocking parity suite, a gap worth closing before it carries more weight.⁴