Perplexity Open-Sources Lily for Faster Qwen Inference on Macs
Perplexity has released the Rust and Metal source, tests and reproduction materials for an engine it says runs Qwen3.6-35B-A3B up to 1.35 times faster than MLX-LM on an M5 Max.
A purpose-built local engine
Perplexity has open-sourced Lily, a compact inference server designed exclusively for the Qwen3.6-35B-A3B model on Apple silicon. The implementation, tests and benchmark reproduction materials are now available in the company’s public pplx-garden repository under the Apache 2.0 license. Lily combines a Rust runtime with Metal kernels compiled from source and exposes a deliberately limited subset of the OpenAI chat-completions interface. PyTorch and Apple’s general-purpose MLX framework are not part of its execution path, although Lily loads a checkpoint converted to MLX affine 4-bit weights.
Lily targets one specific 4-bit checkpoint. Qwen3.6-35B-A3B has 35 billion parameters but activates roughly 3 billion per token, routing each token through eight of 256 experts plus a shared expert. Its checkpoint occupies 19.4 GB, while the model mixes ten full-attention layers with 30 Gated DeltaNet recurrent layers. Perplexity tailored Lily’s data movement, scheduling and kernels around those fixed characteristics.
Faster, within narrow boundaries
On a MacBook Pro equipped with a 40-core M5 Max GPU and 128 GB of unified memory, Perplexity says Lily averaged 1.23 times MLX-LM’s prompt-processing throughput and 1.35 times its token-generation throughput across contexts ranging from 256 to 128,000 tokens. At a 4,000-token prompt and context, it processed prompts at 5,749.9 tokens per second and generated 186.6 tokens per second, versus 4,737.5 and 140.9 for MLX-LM.
The gains came from keeping expert routing and recurrent state on the GPU, fusing operations to avoid intermediate memory traffic, and changing attention paths as context length grows. Perplexity also reported that speculative decoding made this batch-one workload 18% slower because proposed tokens created inefficient execution shapes and scattered expert access.
The comparison remains vendor-run and deliberately narrow, but outside developers can now inspect the implementation and attempt to reproduce the published results. The repository includes dated MLX comparison reports, benchmark harnesses, CPU-reference kernel tests, shader and API tests, and optional end-to-end tests requiring the designated checkpoint. Lily requires an M5 or newer Apple GPU, macOS 26 and the specified quantized Qwen checkpoint; it does not support other Qwen sizes, dense or BF16 checkpoints, or GGUF, AWQ, GPTQ, int8 and fp8 formats. It also supports only text messages and greedy, non-streaming generation, rejecting tools, sampling controls, multimodal input and speculative decoding.
The release matters because local AI performance may increasingly depend on co-designing runtimes for a particular model architecture and device rather than relying entirely on portable frameworks. Lily’s narrow scope limits its immediate audience, but the published code and reproduction materials turn its reported performance from a preview into a result that independent developers can now test.