What it does
llama.cpp aims to make LLM and vision-language model inference possible with minimal setup and strong performance on many kinds of hardware, locally or in the cloud. It is a plain C/C++ implementation with no dependencies, built on the ggml library. It targets Apple silicon through ARM NEON, Accelerate and Metal, x86 CPUs with AVX, AVX2, AVX512 and AMX, and RISC-V. GPU support includes custom CUDA kernels for NVIDIA, HIP for AMD, MUSA, Vulkan and SYCL, and CPU+GPU hybrid inference lets you partly accelerate models larger than the available VRAM. Integer quantization from 1.5-bit to 8-bit reduces memory use and speeds up inference. You can install it with the script from llama.app, run it with Docker, download prebuilt binaries from the releases page, or build from source. Once installed, one command downloads a model from Hugging Face and runs it, and another launches an OpenAI-compatible API server with a built-in web UI.
Capabilities
- Plain C/C++ implementation with no dependencies
- Backends for Apple silicon, CUDA, HIP, Vulkan, SYCL and more
- 1.5-bit to 8-bit integer quantization
- CPU+GPU hybrid inference for models larger than VRAM
- OpenAI-compatible API server with a built-in web UI
- Download and run models directly from Hugging Face
When to use it
- Running open LLMs on your own laptop, server or GPU
- Serving a local OpenAI-compatible endpoint
- Fitting larger models into limited memory with quantization
- Embedding LLM inference in a C/C++ project through the library API
