What it does
Mac LLM Calculator works out whether a large language model fits in the unified memory of a given Apple Silicon Mac. You choose a Mac chip, unified memory size, model, quantization and a context length of 8K, 32K or 128K tokens. The calculator then reports whether the model fits and the expected generation speed in tokens per second. Memory use is parameters in billions times bits per weight divided by eight, plus 15% runtime overhead and the key-value cache for the chosen context, with 3 GB left for macOS. Generation speed comes from memory bandwidth divided by the size of the active weights, plus a fixed cost per token. Macyou says the estimates reproduce its own measurements to within 4.6%. Macyou says every row outside the M4 and the M5 Pro is calculated, not measured.
Capabilities
- A memory bandwidth table covers Apple Silicon chips from M1 to M6 with memory options and the Mac models each chip is found in.
- A per-model table at Q4_K_M lists weights, minimum memory and generation speed on M4, M4 Pro, M4 Max (40-core GPU) and M5 Ultra.
- Mixture of experts models are handled separately, with total parameters deciding memory fit and active parameters deciding speed.
- Results include an option to rent a dedicated Mac, for example an M4 Mac mini with 24 GB at $113 /mo billed annually or $143 /mo billed monthly.
- The prediction model with its data and tests is on GitHub under MIT.
When to use it
- You are choosing which Mac to buy to run a language model locally.
- You want to check whether a model fits in unified memory at a given context length.
- The model you want needs 64 GB or more and the machine has to be reachable around the clock.
