TL;DR
- Moving individual operations to the NPU made decoding slower. In the first matched test, Vulkan alone reached 102.93 tokens/s; Vulkan with one feed-forward block on XDNA1 reached 38.28.
- A purpose-built 6-bit decoder did work. The full 24-layer Qwen2.5-0.5B decoder used 46.4% fewer packed weight bytes and was 1.298× faster than the matched BF16 NPU decoder. It passed our held-out quality screen with 1.0065× perplexity and 470/512 matching top predictions.
- That did not make it the better standalone engine. The Radeon produced the first token in 49–94 ms across the tested prompts, versus 1.15–3.00 s for W6. With its 64-token rolling context, W6 returned only a ten-token fragment on the longest prompt.
- Drafting helped on predictable text, not everywhere. W6 24K sped up a JSON continuation by 1.817× versus GPU-only, but CPU drafting led at 1.949× in that same comparison. A CPU lookup → NPU draft → GPU verification follow-up worked correctly and still lost to the simpler choices.
There is a tempting question when a laptop has a CPU, an integrated GPU, and an NPU: why not use all three to make a local language model faster? We tried to answer it on a Ryzen 9 8945HS with a Radeon 780M and AMD’s first-generation XDNA NPU. The work started with a GGML offload plugin and ended with a complete, quantized decoder built for that NPU. The most useful result was finding exactly where the extra hardware helped—and where it didn’t.
1. The offload plugin worked. It still lost.
We started with an external GGML plugin, wired into an experimental Lemonade recipe. A GGUF is a weight container, not something an NPU can execute directly. The plugin had to decode supported weights, repack them into its device layout, and dispatch compiled AIE kernels. A chat smoke test passed, and the counters confirmed the NPU was doing work. Neither fact answered whether it was doing that work faster.
An early CPU+NPU chat request took about 49 seconds to emit nine tokens. That included cold conversion, so it wasn’t a fair estimate of warm inference. The useful comparison used Qwen3 0.6B Q4_0 and three controls: CPU-only, Vulkan-only, and Vulkan with the feed-forward work from one transformer block reserved on the host. We then put that same block on XDNA1. The host/CPU partition itself had a cost; moving it to the NPU cost more. The NPU partition reached about 37% of full Vulkan decode speed and about half the speed of the matched host/CPU partition. Those were short decode runs, not a sustained benchmark, but the regression was too large to ignore.
Bar lengths are scaled to the largest value shown.
Median decode (tok/s)
Vulkan only: 102.93 tok/s
CPU only: 83.17 tok/s
Vulkan + host/CPU block: 75.77 tok/s
Vulkan + XDNA1 block: 38.28 tok/s
The shape mismatch explained why we shouldn’t expect a quick fix. A small dense model was being mapped to padded 2,048-by-2,048 weight tiles. Counting eligible weight slots suggested roughly 17 times the actual weight positions and a potential working set far beyond the plugin’s 512 MiB cache. That was a layout estimate, not a cache trace. Still, increasing the cache would not remove padded computation or the cost of crossing a device boundary for individual operations. It was time to change the unit of work.
2. Put a whole small decoder on the NPU
Instead of borrowing the NPU for pieces of a larger model, we compiled a small, architecture-specific Qwen2.5-0.5B-Instruct decoder. It executes 24 layers and the output head in one NPU dispatch per token. This particular graph had a 64-token cache; the host slides it and rebases cached RoPE keys once the window fills.
That made it a candidate for speculative decoding. The small model proposes a continuation, and a larger Qwen3.5 9B model on Vulkan checks several proposed tokens in one batch. The target decides what gets emitted. A fast drafter only helps if it proposes enough tokens the target accepts to repay the extra work. We saw a median 1.64× total speedup for an early JSON counting continuation and 1.46× for counting, but the same approach did not generalize to coding and other prompts.
Before building W6, we tried a smaller output head with BF16 weights. A 49K-token shortlist fell just short of our 1.10× end-to-end gate on JSON. A 24,576-token shortlist passed: 16.23 versus 14.64 decode tokens/s and a 1.103× paired total speedup over full-vocabulary NPU drafting in that session. The shortlist never constrained the target’s vocabulary, and a quantized CPU drafter was still faster. Nor was target-token agreement something we could assume: an earlier fixed lookup control diverged on all three paired coding runs despite using greedy verification. Different GPU batch shapes can change near-tied predictions, so subsequent performance claims required exact emitted token agreement with the stable GPU-only control.
Profiling pointed us away from the obvious fixes. A tiny dispatch took about 0.085 ms, compared with about 26 ms for a BF16 shortlist token. That control included small-kernel work, so it was not an exact launch-overhead subtraction, but it gave no reason to blame launch alone. Profiling the real graph showed a median 76.85% of the measured stage time in the MLP and down projection together, with gate/up projection calls dominating the profiled MLP. A weight-stream control gave a useful comparison, not proof of a bandwidth ceiling. A W8 reconstruction path also shrank the weight stream without speeding up the decoder. And a supposed four-deep prefetch experiment turned out to allocate only two physical consumer buffers. We corrected the claim after inspecting the lowered graph.
3. Four bits won the kernel benchmark and failed the model
Our first packed W4 kernels preserved the intended numerical results, but unpacking and reconstructing BF16 weights made them slower than BF16. A row-wise power-of-two W4 format with integer accumulation changed that: on six workers, the long-stream dot and down controls gained about 1.42× and 1.55×, respectively. We could now run a fast projection. Could we trust the model built from it?
| Candidate | Split | Perplexity ratio | BF16 top-prediction agreement |
|---|---|---|---|
| Original BF16 weights, INT16 activations | Test | 0.996× | 98.24% |
| Row-wise W4, INT16 activations | Test | 11.088× | 23.05% |
| Train-calibrated group-32 W4 | Validation | 1.152× | 75.20% |
| Train-calibrated group-32 W6, deferred scaling | Validation | 1.00519× | 93.16% |
Row-wise W4 drove windowed perplexity from 27.13 to 300.87. The activation-only ablation stayed close to baseline, which pointed to coarse weight quantization rather than the INT16 conversion. Grouping and protecting more layers helped W4, but not enough to clear both thresholds: at most 1.10× perplexity and at least 90% agreement with BF16’s top next-token choice.
W6 was the first grouped candidate to survive. We fitted scales with input activation statistics from the training split, then screened on separate WikiText-2 validation windows. The calibration used 512 activation observations per projection, across 97 projections. It was a diagonal weighted fit with BF16 scales for groups of 32 weights, not a full Hessian method or a fine-tune. All screens fed the same ground-truth tokens to each candidate and evaluated the full vocabulary; they did not let models generate their own context. Passing that CPU screen chose a model representation, not an NPU speed winner.
4. The fastest way to store six bits was, for this test, a byte
A four-row-by-896 block used 7,168 bytes at BF16 versus 2,944 bytes in packed W6, including scales and padding. The first packed W6 path reconstructed BF16-rounded weights correctly, yet ran roughly 12× slower than BF16 in the six-worker streamed comparison. Moving scale application after the integer group dot—deferred scaling—removed much of that work. It also changed the rounding semantics, so we screened it separately. The result still ran about 3× slower than BF16.
Then we tried using one signed byte per W6 code. The byte block grew to 3,840 bytes, but it kept the same six-bit values and BF16 scale bits. It got rid of bit-plane unpacking and was about 1.9× faster than packed deferred W6, with exact agreement against that control. It was still slower than BF16. Smaller storage helped only when the device could consume it efficiently.
The later wins came from changing the data layout around the math. Four-row reduction reused work across rows. A four-group tile replaced 112 scalar group-dot stores with 28 128-bit stores per block; a separate unroll-only control confirmed the gain was not just loop unrolling. Paired-row computation reduced activation shuffles, and pre-interleaving immutable weight bytes gave the device operands it could load directly. That last layout gave a 1.14–1.15× gain over the qualified four-row tile and began passing six-worker down-projection comparisons against BF16. Dot projections were less consistent.
Some plausible optimizations did not pay off. Unsigned-limb scaling missed the strict bit-equivalence check despite small mathematical errors. The corrected integer-residual path was exact but no faster. Outlining a proof helper changed the diagnostic timings because inline proof code caused stack stores. Preparing doubled activations once per dispatch reduced hot-loop shuffles, then added cache traffic and delivered only a 1.016–1.033× gate/up slope gain. Counting instructions was useful for explaining a result, not for declaring a win.
The bigger improvement came from using the NPU’s native INT16×INT8 matrix instructions for dot projections. We changed how weights were arranged, not their codes or scales. On held-out real projection replay, matrix-dot passed 18/18 BF16 timing gates in the first sampled dot configuration set. A follow-up across all 24 layers passed 47/48 BF16 gate/up comparisons. Then we fused gate/up, SiLU, and row-pair down on device, so the down projection consumed the live intermediate. The fused MLP passed all three sampled layer comparisons at 1.44–1.51× paired slope speedups. That, rather than a synthetic matrix benchmark alone, justified building the complete decoder.
5. The full W6 decoder fit—with 96 bytes to spare
The final prototype used matrix-dot for QKV, attention output, gate/up, and the full 151,936-entry output head. Down used native row pairs. Embeddings, norms, biases, and the K/V cache stayed at their original precision. The compact vector-RMSNorm build was still too big for a worker’s 16 KiB program memory. Sharing conversion and scaling code, using a compact scalar RMSNorm, and reusing a BF16 copy path brought the largest worker program down to 16,288 bytes.
Bar lengths are scaled within each group.
Median decoder step (ms/token)
BF16 NPU: 35.97 ms/token
W6 NPU: 27.95 ms/token
Packed weights (decimal MB)
BF16 NPU: 1011 MB
W6 NPU: 542 MB
The decoder passed the actual hardware gate on 512 scored test positions: 27.178 windowed perplexity versus 27.001 for the matched BF16 NPU graph, and 470 matching top predictions out of 512. Thirteen of sixteen individual windows passed both limits. Across 6,144 hardware dispatches, scored logits repeated bit-for-bit between repetitions of the same mode. The packed weight sizes in the chart exclude the unchanged token embedding. The 1.298× paired token-latency result includes device and host step costs but does not include model loading, and it is not standard full-corpus perplexity.
6. The user-facing benchmark changed the answer
Teacher-forced token latency tells us whether an engine got faster; it doesn’t tell us whether somebody gets a better answer sooner. For that, we used identical ChatML token inputs and greedy generation to compare the same Qwen2.5-0.5B checkpoint against llama.cpp on CPU and Vulkan. The NPU processed prompt tokens serially, with a 64-token rolling cache. The CPU and GPU used their native batched prefill and 512-token context, with prompt caching disabled. We intentionally measured the usable implementations rather than pretending they had identical context semantics.
Bar lengths are scaled within each group.
Time to first token (ms)
JSON (45 input tokens) · BF16 Vulkan: 82 ms
JSON (45 input tokens) · W6 NPU: 1202 ms
Short Python (43) · BF16 Vulkan: 49 ms
Short Python (43) · W6 NPU: 1146 ms
Long Python (108) · BF16 Vulkan: 94 ms
Long Python (108) · W6 NPU: 3005 ms
Decode after first token (tok/s)
JSON (45 input tokens) · BF16 Vulkan: 47.3 tok/s
JSON (45 input tokens) · W6 NPU: 35.4 tok/s
Short Python (43) · BF16 Vulkan: 48.7 tok/s
Short Python (43) · W6 NPU: 35.0 tok/s
Long Python (108) · BF16 Vulkan: 48.6 tok/s
Long Python (108) · W6 NPU: 34.5 tok/s
In the first standalone run, W6 decoded slightly faster than four-thread BF16 CPU—about 35 versus 31–33 tokens/s—but serial prefill gave W6 a much later start. Once Radeon Vulkan was available, it won both time to first token and decode rate on all three prompts. The long Python prompt had 108 input tokens, so the NPU rolled its cache 44 times during prefill. BF16 NPU stopped immediately; W6 returned a ten-token fragment. The Radeon ran to the 128-token cap, which is not the same thing as completing the task. An engine that emits a shorter or empty answer has not become faster at completing the request.
Getting both devices into the same environment was its own lesson. Recreating the test container meant restoring Vulkan and XRT libraries, exposing the Radeon render device, and raising an 8 MiB locked-memory limit. XRT needed a 64 MiB mapping just to initialize. We verified actual Radeon offload rather than accepting a software renderer or assuming that a Vulkan label meant the GPU was working.
7. Did all three processors help together?
The W6 24K draft package copied the decoder layers unchanged and retained only selected output-head rows, bringing the W6 weight stream from 542 to 420 MB. On 96 inputs, including 32 cache rollovers, the selected logits and K/V bits matched the full W6 engine exactly. In the first paired W6 drafting comparison against the same Radeon target, W6 24K reached a 1.817× median total speedup on JSON versus GPU-only. BF16 24K reached 1.678×; CPU drafting led at 1.949×. Directly paired against BF16, W6’s median gain was only 1.083× and it lost one pair, so it didn’t clear the replacement gate. All 18 drafted runs matched their paired GPU-only outputs. All drafters fell back and lost on coding in that session. These were comparisons of usable implementations: the CPU draft used four threads, Q4_K_M weights, and a much longer context than the NPU.
We also tried giving the CPU and NPU distinct proposal jobs in the same request. The CPU looked for a previous token suffix with a known continuation. Only a miss could call the W6 drafter; the GPU verified either source. We disabled NPU lookahead for this variant, because speculative NPU work could otherwise start before we knew whether the CPU lookup would hit. The experiment included CPU lookup alone, ordinary W6 with overlap, CPU neural drafting, and GPU-only. The CPU neural drafter could adjust its proposal length; the NPU and lookup modes proposed up to three tokens at a time. When drafting cost more than it saved, the benchmark could fall back to GPU-only and kept those runs in the results.
Bar lengths are scaled to the largest value shown.
Median paired total speedup
GPU only · JSON continuation: 1.000×
GPU only · Coding prompt: 1.000×
CPU neural draft · JSON continuation: 2.840×
CPU neural draft · Coding prompt: 1.405×
CPU lookup only · JSON continuation: 1.005×
CPU lookup only · Coding prompt: 0.959×
W6 NPU draft · JSON continuation: 1.922×
W6 NPU draft · Coding prompt: 0.959×
CPU lookup → W6 · JSON continuation: 1.477×
CPU lookup → W6 · Coding prompt: 0.966×
On JSON, the GPU accepted none of the 30 CPU lookup proposals per hybrid run. The hybrid still beat GPU-only, thanks to the NPU, but lost to both plain W6 and CPU drafting. It cut NPU dispatches from 199 to 165 per run, while GPU verification rounds rose from 38 to 45. On coding, both sources contributed real tokens: the GPU accepted three lookup tokens and 27–53 NPU draft tokens per run. Even so, the hybrid was slower than GPU-only in every coding pair. All 24 drafted runs matched their paired GPU-only output token for token. The CPU neural drafter helped coding in this follow-up, unlike the earlier four-way session; neither session alone supports a general coding policy. And because the hybrid deliberately gave up lookahead, the results compare usable policies, not the isolated cost of lookup.
The implementation we kept
- General generation: ordinary Vulkan, no draft by default. No new Lemonade server route or default NPU backend.
- Repeatable, high-acceptance work: explicit CPU drafting. The measured JSON continuation was a good fit. Arbitrary JSON, coding, and general chat are not automatically routed to a drafter.
- NPU development: retain the pinned W6 decoder and 24K draft package. They are working experimental artifacts, not a generic GGUF import path or production service.
We also made the standalone experiment CLI default to GPU-only when no draft is selected. Existing commands that deliberately supply an NPU draft still work. In the handoff check, the new default and the explicit CPU option both reproduced all 128 target tokens from the qualified Vulkan reference without opening an NPU context for inference.
What I would carry into the next accelerator project: test the model’s output before integrating a faster kernel; inspect the compiled layout, not just the requested settings; and compare against the fastest path someone can actually use. The 6-bit decoder was a real engineering win. The measurements also kept us from mistaking it for the right default.
Reproducibility and limits
- Hardware: Ryzen 9 8945HS, Radeon 780M, and first-generation XDNA1. The closing Vulkan comparisons used RADV/Mesa 26.2.7 and XRT 2.21.75, package revision 9.34, on Linux 7.3.4.
- Dependencies and data: hawkpoint-npu-llm at
64c5ffcdb15babd009137397b531e63005b16198, llama.cpp b10091, Qwen2.5-0.5B-Instruct, and WikiText-2 raw. The larger GPU target used the pinned Qwen3.5 9B Q4_K_XL GGUF. - The experiment record lives under
contrib/xdna1/in the Lemonade working branch. Key milestones are991acc411(full W6 decoder),3c6f35062(Vulkan comparison),616f81883(final implementation decision), and38d72c544(three-device follow-up). - These are results for one system, pinned models, selected prompts, and greedy decoding. Energy, multi-request throughput, longer native NPU context, arbitrary GGUF compatibility, and production HTTP integration were not qualified.
Leave a Reply