One C++ binary. Zero Python. MIT licensed.
Including on a chip nothing else can touch: AMD's own runtime for the Strix Halo NPU is closed-source, so we reverse-engineered it from scratch. Also runs on AMD/NVIDIA/Apple GPUs and plain CPU. It's an inference engine, not a chat app — bring your own frontend. 47 models run out of the box, 135M to 74B parameters.
We converted these to 1BP — our own single-file, zero-config, memory-mappable format — with a pure-C++ toolchain (no Python). Every one is published on Hugging Face and runs end-to-end on this engine. Numbers below are measured on-device (Strix Halo).
Zaya1-8B is the model this project was built around — the first validated end-to-end through Q4NX, GGUF, and 1BP. The same auto-detect pipeline now covers 17 families and 47 1BP models (+12 documented Zyphra non-LLM models), from dense transformers to MoE to state-space hybrids. Full specs for every family live in the catalog on GitHub →
16-expert MoE. Every tensor structurally and numerically verified against the source GGUF — the first target of every new converter this project ships.
18 architectures — auto-detected, no config needed
+ 15 more families · 40+ more 1BP models + documented Zyphra non-LLM
Every family above converts to the same single-file, memory-mappable 1BP format — zero external config.json or tokenizer.json. One of them skips conversion entirely: these run straight from GGUF, validated end-to-end.
| Model | Params | Backend | Tests |
|---|---|---|---|
| Qwen3-0.6B | 0.6B | ZINC · NPU | ✅ 28/28 |
| Qwen3-VL-4B | 4B | ZINC | ✅ 36/36 |
| Llama-3.1-8B | 8B | ZINC · NPU | ✅ 32/32 |
| Qwen3-8B | 8B | ZINC · NPU | ✅ 36/36 |
| Gemma4-E2B | 2B | ZINC · NPU | ✅ 35/35 |
Every number above and on the model cards traces back to one authoritative file (site/benchmarks.json) with a status tag — validated / optimized / raw — so nothing here is cherry-picked. Kernel-level numbers are isolated GEMV throughput, not end-to-end decode; the full breakdown below keeps the two apart.
Single binary routes every layer to the fastest available accelerator. Auto-detects hardware at startup — no config, no restart.
npu_xrt — no closed-source subprocess.No Python. No pip. No Docker. One binary runs any model.
Package formats: deb · snap · AppImage · Homebrew · AUR · Docker · Ollama · more →
No. Pure C++23. No Python, no pip, no Docker at runtime. The single binary is all you need.
AMD Strix Halo (Ryzen AI Max+ 395) — the only laptop APU with XDNA 2 NPU (50 TOPS) + RDNA 3.5 GPU in one chip. Also runs on any AMD system with ROCm or Vulkan.
No. Built from scratch in C++23. The only thing it shares with llama.cpp is that both read GGUF files. We reverse-engineered AMD's NPU stack from binary — 22 proprietary .so files, 209 xclbin bitstreams — and rebuilt it open source.
MIT. View license →
llama.cpp is faster on the same hardware (229 tok/s vs our 30-80 tok/s). We say so in the README. Our advantage is NPU support, model-agnostic auto-detection, and zero Python — all in a single binary. See issue #235 for discussion.
Known issue: Linux 6.19.x kernels have a reproducible amdgpu OPTC CRTC hang under sustained NPU/GPU load on Strix Halo (gfx1151). The display pipe locks up mid-inference. Confirmed-stable: 6.18.22-lts and 7.x. Run uname -r to check yours. The install script warns on 6.19.x automatically.
CMake ≥ 3.28, Ninja ≥ 1.12, GCC ≥ 13 (C++17), and ROCm's amdclang++ for HIP kernels (shipped with TheRock 7.1.5a). The install script handles all of this automatically.