What it does
jitllm describes itself as an operating system for large language model inference. Each program is one self-contained binary for Linux, macOS and Windows on x86-64 and arm64. GPUs need only their driver. jitllm detects the CPUs and GPUs on the machine, generates kernels for them, and places each block of a model on a device at runtime. Models larger than memory page from disk. Layers move between devices without losing the conversation. To start, convert GGUF or Hugging Face weights once into the .jlm container, which preserves quantization. Then serve the model with jitllmd or run it from the command line with jitllm. The jitllm-desktop app adds downloads, conversion, chat and a live view of the blocks. The engine also embeds as a Go library.
Capabilities
- jitllmd serves the OpenAI-compatible endpoint POST /v1/chat/completions and the Anthropic-compatible endpoint POST /v1/messages. A client is pointed at http://localhost:4242/v1 and uses the model's id as its name.
- The Connect API is a control plane for loading and unloading models, creating, parking and resuming sessions, and moving blocks between devices. It also provides /metrics for Prometheus.
- Continuous batching runs on the CPU and the GPU with KV-aware admission. The -fairness 0-100 option time-slices sessions through the card.
- Structured output works from a JSON schema, with log-probabilities and n choices from one prefill. tool_choice is enforced by constrained decoding.
- Speculative decoding, an int8 KV cache and prefix caching are included.
When to use it
- You want to run a model that is larger than memory by paging its weights from disk.
- You need to serve models to clients written against OpenAI or Anthropic APIs.
- You want to embed an inference engine inside your own Go program.
