Hypura is a new LLM inference scheduler for Apple Silicon that distributes models across GPU, RAM, and NVMe.

Source Brief

Hypura is a new LLM inference scheduler for Apple Silicon that distributes models across GPU, RAM, and NVMe. This enables running models larger than physical memory without system crashes. A 31 GB Mixtral model can run on a 32 GB Mac Mini using this technology.