Perplexity Announces Lily for Faster Qwen Inference on Macs
The specialized Rust and Metal engine runs Qwen3.6-35B-A3B up to 1.35 times faster than MLX-LM on an M5 Max, according to Perplexity.
A purpose-built local engine
Perplexity has announced Lily, a compact inference server designed exclusively for the Qwen3.6-35B-A3B model on Apple silicon, and says it will open-source the engine soon. Lily combines a Rust runtime with custom Metal kernels and exposes a minimal OpenAI-compatible chat-completions interface. PyTorch and Apple’s general-purpose MLX framework are not part of its execution path.
Lily currently accepts one specific 4-bit checkpoint. Qwen3.6-35B-A3B has 35 billion parameters but activates roughly 3 billion per token, routing each token through eight of 256 experts plus a shared expert. Its checkpoint occupies 19.4 GB, while the model mixes ten full-attention layers with 30 Gated DeltaNet recurrent layers. Perplexity tailored Lily’s data movement, scheduling and kernels around those fixed characteristics.
Faster, within narrow boundaries
On a MacBook Pro equipped with a 40-core M5 Max GPU and 128 GB of unified memory, Perplexity says Lily averaged 1.23 times MLX-LM’s prompt-processing throughput and 1.35 times its token-generation throughput across contexts ranging from 256 to 128,000 tokens. At a 4,000-token prompt and context, it processed prompts at 5,749.9 tokens per second and generated 186.6 tokens per second, versus 4,737.5 and 140.9 for MLX-LM.
The gains came from keeping expert routing and recurrent state on the GPU, fusing operations to avoid intermediate memory traffic, and changing attention paths as context length grows. Perplexity also reported an instructive failure: speculative decoding made this batch-one workload 18% slower because proposed tokens created inefficient execution shapes and scattered expert access.
The comparison remains vendor-run and deliberately narrow. Perplexity says Lily requires M5-class hardware, macOS 26 and the designated quantized Qwen checkpoint; it does not support other Qwen sizes, dense models or common GGUF, AWQ and GPTQ formats. Because the code has not yet been publicly released, outside developers cannot currently reproduce the benchmark or inspect the implementation.
The announcement matters because local AI performance may increasingly depend on co-designing runtimes for a particular model architecture and device, rather than relying entirely on portable frameworks. Lily’s reported results suggest a substantial advantage, but its limited compatibility and pending code release leave open whether those optimizations can become a durable platform rather than a one-model showcase.