FreeToken: Edge-Native MoE Serving Engine for Consumer Hardware

24 August 2026 2 min read

#consumer-gpu #deepseek #glm #memory-efficient-inference #memory-optimization #mixture-of-experts #moe #quantization #release #zhipu

There is a full guide to the arithmetic underneath this

MoE Expert Offload: What a 35B Model Actually Costs on a 12GB Card — why most of an MoE model can live outside VRAM at all, the decode roofline that expert caching and elastic KV reallocation are each moving one term of, and how to work the same sums for your own card.

FreeToken is a mixture-of-experts (MoE) serving engine built for edge-native inference — running frontier-scale open-weight models on personal and consumer hardware rather than on cluster infrastructure. Its stated target is 290B+ parameter MoE models on a gaming PC, and the supported list includes GLM-5.2, DeepSeek-V4-Flash and Qwen3.6-35B-A3B.

The approach is not single-GPU. FreeToken treats heterogeneous edge resources — GPUs, CPUs, host memory and interconnects — as one elastic pool, with bandwidth-adaptive CPU–GPU co-execution, expert caching, and dynamic reallocation of VRAM between expert caches and KV memory at runtime.

The project publishes a paper (arXiv 2608.16157) describing the bandwidth-adaptive execution scheme.

Read the full article on Google News.

→ MoE Expert Offload: What a 35B Model Actually Costs on a 12GB Card works through why a claim like "290B+ on a gaming PC" is arithmetically possible at all: in an MoE, what must be stored and what must be read per token are different numbers, and only the second one sets your ceiling. That is also the sum that decides where expert caching and dynamic VRAM-versus-KV reallocation stop helping — the KV cache term grows with context until it dominates all expert traffic combined. FreeToken's own throughput claims are unverified here.

Source: Google News · Relevance: 9/10