FreeToken
#8 in Local LLM RuntimesUnknown · since 17. August 2026 (arXiv-Preprint und Open-Source-Release) · 2× · last seen Aug 26, 2026
FreeToken is an open-source inference engine by FlashML (with researchers from UC Berkeley, UT Austin and MIT) for running large Mixture-of-Experts models locally on consumer hardware. The software dynamically distributes model weights between GPU VRAM, system RAM and CPU, enabling models such as Qwen3.6-35B, DeepSeek-V4-Flash 284B or GLM-5.2 753B to run on a single consumer or workstation GPU with limited VRAM. It offers OpenAI- and Anthropic-compatible API endpoints for integration with coding agents like Claude Code, Codex or OpenCode. The source code is available under Apache 2.0 license on GitHub, with an additional desktop app for Windows and Linux and a PyPI package.
Features
| Throughput/Latency | Qwen3.6-35B: 39.3 tok/s on 8GB RTX 4060 laptop; DeepSeek-V4-Flash 284B: 22-25 tok/s on RTX 5090; GLM-5.2 753B: 14.9 tok/s on RTX PRO 6000; 2-4x faster than Ollama |
| License | Apache License 2.0 |
| Platform | Linux x86_64 (CLI) and Windows/Linux desktop app; NVIDIA GPU (RTX 30/40/50 series), CUDA 13, driver r580+ |
| Price | Free/open source (only own hardware costs) |
| Protocol Compatibility | OpenAI-compatible API (/v1/chat/completions, /v1/responses, /v1/models) and Anthropic-compatible API (/v1/messages) |
| Release Date | August 17, 2026 (arXiv preprint, open-source release) |
| Supported Models/Providers | 20+ MoE models incl. DeepSeek-V4-Flash, Qwen3.6-35B-A3B, GLM-5.2; formats MXFP4, NVFP4, FP8, BF16 |