What it does
AirLLM is an open-source Python library, installed with pip, for running large language models on hardware with little GPU memory. Instead of holding a whole model in memory, it keeps a single layer on the GPU at a time, so the memory needed depends on the size of a layer rather than the size of the model. Models are loaded through one AutoModel interface by passing a Hugging Face repo ID or a local path. The project says this works with most popular open model families, including Llama, Qwen, DeepSeek, Mistral, Mixtral, Phi, Gemma, ChatGLM, Baichuan, InternLM and Yi. Its own table lists a 70B Llama 3.x model at about 4 GB of VRAM, Llama 3.1 405B at about 8 GB and DeepSeek-V3 671B at about 12 GB. Optional 4-bit or 8-bit block-wise compression shrinks the weights that are loaded from disk, which the project names as the main bottleneck. Newer releases add LoRA-style training in which frozen weights stream from disk one layer at a time while the adapters stay on the GPU. The first run splits the original model into per-layer files, so plenty of disk space is needed. The project also notes CPU inference and macOS support on Apple silicon.
Capabilities
- Single AutoModel entry point that takes a Hugging Face repo ID or local path
- Keeps one layer on the GPU at a time to reduce VRAM needs
- Optional 4-bit or 8-bit block-wise compression
- Layer-by-layer LoRA training with adapters kept on the GPU
- Runs on Linux and on Apple silicon Macs
- CPU inference support
When to use it
- Running a model that is far larger than your GPU memory would normally allow
- Trying newly released open models through the same few lines of loading code
- Training LoRA adapters for a large model on a small GPU
