What it does
LocalAI is an MIT-licensed runtime for running AI models on hardware you control. It exposes an API that existing OpenAI clients can talk to, and it also speaks the Anthropic, Ollama and ElevenLabs APIs, so most applications only need a changed URL. Language models, speech recognition, speech synthesis with voice cloning, vision and detection, image, video and music generation, and agent workloads can all run through the same endpoint. Different inference engines such as llama.cpp, vLLM, SGLang and MLX can sit behind one endpoint, chosen in a model's config, and engines are pulled in only when a model needs them. The project says every feature ships with a tested CPU path, and it lists support for CUDA, ROCm, SYCL, Metal and Vulkan, plus Docker and Kubernetes. A distributed mode adds routing, placement based on free VRAM, autoscaling and failover across machines. The site also describes a gallery of 1,255 models, quantised builds it calls APEX, and a built-in agent started with the local-ai chat command that can load MCP servers, plugins and skills.
Capabilities
- OpenAI-compatible API that also speaks the Anthropic, Ollama and ElevenLabs APIs
- Text, speech, vision, image, video and music workloads behind one runtime
- Swappable engines including llama.cpp, vLLM, SGLang and MLX
- A CPU path for every feature, with optional GPU acceleration
- Distributed mode with VRAM-aware routing, autoscaling and failover
- Model gallery of 1,255 models and a built-in agent with MCP support
When to use it
- You want an existing OpenAI-style application to run against models on your own machines.
- You need to keep data on your own hardware, including on machines without a GPU.
- You want one local service for chat, transcription, speech and image generation.
- You plan to spread model serving over several machines.
