What it does
bitnet.cpp is a framework for running 1-bit large language models, such as BitNet b1.58, on your own hardware. It ships optimized kernels for CPU inference and has released a GPU inference kernel as well, while NPU support is listed as coming next. The project publishes its own measurements against full-precision models: speedups of 1.37x to 5.07x on ARM CPUs and 2.37x to 6.17x on x86 CPUs, with energy use reduced by 55.4% to 70.0% on ARM and 71.9% to 82.2% on x86. It also says a 100B BitNet b1.58 model can run on a single CPU at roughly human reading speed, around 5 to 7 tokens per second. The repository is built on llama.cpp. It includes scripts to set up a model, run inference with an optional chat mode, benchmark throughput, and convert .safetensors checkpoints to GGUF. Supported models include the official BitNet-b1.58-2B-4T language model (2.4B parameters) and two 1-bit embedding models, plus community models such as the Falcon3 and Falcon-E families. Building from source needs Python 3.10 or newer, CMake 3.22 or newer and clang 18 or newer.
Capabilities
- Optimized kernels for fast 1.58-bit inference on CPU, plus an official GPU kernel
- Chat mode for instruction-tuned models through the run_inference.py script
- Official 2.4B-parameter BitNet-b1.58-2B-4T model and 0.6B and 270M 1-bit embedding models
- Benchmark script and a converter from .safetensors checkpoints to GGUF
- Built on llama.cpp and released under the MIT license
When to use it
- You want to run a 1-bit language model on a laptop or server CPU
- You are measuring speed or energy use of ternary-weight models on your own machines
- You want to try the 1-bit embedding models for CPU-based inference
Alternatives to explore
- T-MAC
