What it does
Speech To Speech is an open source toolkit for building voice agents from open models. One command starts a spoken conversation on your own computer with no API key or cloud service, and once the first run has downloaded the models it works offline. The pipeline has four stages: voice activity detection, speech to text, a language model, and text to speech. The default combination uses Silero VAD, Parakeet TDT, Qwen3.5-4B and Qwen3-TTS, and every stage can be swapped for other backends. The language model can run in process, on a separate server such as llama.cpp or vLLM, or at a hosted provider, while speech recognition and synthesis stay local. A server speaks the OpenAI Realtime protocol over WebSocket and WebRTC, so apps written for that API can point at your own machine. The page notes this covers a tested core subset of the events, not the full API. It says the pipeline runs in production as the conversation backend for thousands of Reachy Mini robots, and optional viseme output adds mouth shapes for avatar and robot clients. For the default local setup, plan on 16 GB of unified memory on Apple Silicon or 24 GB of VRAM on NVIDIA.
Capabilities
- Runs locally with no API key, and offline after the first run
- Swappable voice detection, speech-to-text, language model and text-to-speech stages
- OpenAI Realtime-compatible WebSocket and WebRTC server
- Local, self-hosted or hosted language model backends
- Optional viseme output for avatars and robots
- Tool calling support in the packaged client
When to use it
- You want a private voice assistant that runs on your own hardware
- You are building an app or robot against an OpenAI Realtime-style API
- You want to mix and match open speech and language models
- You want speech to stay local while a hosted model handles the replies
