What it does

Petals lets you load a small part of a large model on your own machine while other people in the network serve the remaining parts. The README gives examples with Llama 3.1 (up to 405B), Mixtral (8x22B), Falcon (40B+) and BLOOM (176B), and shows a Python snippet that uses a Hugging Face Transformers-style class to generate text. You can also fine-tune models, use your own sampling methods and inspect hidden states, with the comforts of an API and the flexibility of PyTorch. Reported single-batch speeds are up to 6 tokens per second for Llama 2 (70B) and up to 4 for Falcon (180B). A Colab notebook and a chatbot web app are offered for trying it. In the public swarm your data is processed with the help of other people, so for sensitive data the README suggests a private swarm among people you trust. Petals is community-run and relies on people sharing GPUs, and hosting instructions cover Linux, Windows with WSL, Docker and macOS on Apple silicon. Some models, such as Llama, require requesting access to the weights first. The project is MIT licensed.

Capabilities

Runs very large models by sharing layers across a volunteer GPU network
Transformers-style Python interface
Fine-tuning, custom sampling and access to hidden states
Private swarm option for sensitive data
Hosting setups for Linux, Windows with WSL, Docker and macOS

When to use it

You want to experiment with a very large open model without hardware that can hold it
You have spare GPU capacity to contribute to a shared swarm
You need fine-tuning or hidden-state access rather than text from a hosted API alone