What it does
FastChat is an open-source repository from lm-sys that its README calls an open platform for training, serving and evaluating large language model chatbots. It is the release repository for Vicuna and the code behind Chatbot Arena, which the README says has served over 10 million chat requests across 70+ models and collected more than 1.5 million human votes from side-by-side battles. For running models it provides a command-line chat client, a distributed serving setup of a controller, model workers and a Gradio web server, and OpenAI-compatible REST APIs that can act as a local drop-in replacement. The README lists many supported models, including Llama 2, Vicuna, Falcon and ChatGLM, and says API-based models from OpenAI, Anthropic, Gemini and Mistral can be served too. Instructions cover single and multiple GPUs, CPU only, Apple Silicon, Intel XPU and Ascend NPU, with 8-bit compression for tight memory. For evaluation it includes MT-bench, a set of multi-turn questions where strong LLMs act as judges. For training it includes fine-tuning code based on Stanford Alpaca with multi-turn support, documented for local GPUs, with LoRA instructions in its docs and cloud runs through SkyPilot.
Capabilities
- Training, serving and evaluation code in one repository
- Distributed multi-model serving with a Gradio web UI
- OpenAI-compatible REST APIs for supported models
- Command-line chat on GPU, CPU, Apple Silicon, Intel XPU and Ascend NPU
- MT-bench evaluation with LLMs acting as judges
- Fine-tuning code for Vicuna-style models, with LoRA and SkyPilot cloud options
When to use it
- You want to self-host open models behind an OpenAI-compatible API
- You want a Chatbot Arena-style side-by-side comparison interface for your own models
- You want to benchmark chat models with MT-bench
- You want to fine-tune a Llama-based chat model on your own conversation data
