Mixtral 8x7B

#100 in Open-Source LLMs

Mistral AI · 8× · last seen Jun 30, 2026

6

Momentum

Mixtral 8x7B is an Open Source language model developed by Mistral AI featuring a sparse Mixture-of-Experts architecture (SMoE). Each layer consists of eight expert Feedforward blocks, of which only two are actively used per Token, resulting in approximately 12.9 billion parameters being activated per Token despite a total of around 46.7 billion parameters. The model was released on December 11, 2023 under the permissive Apache 2.0 license and, according to Mistral AI, outperforms Llama 2 70B on most Benchmarks at approximately six times faster Inference, while matching or exceeding GPT-3.5 on many standard Benchmarks. It supports a context window of 32,000 Token and is proficient in English, French, German, Spanish, and Italian; additionally, an instruction-tuned variant exists.

Sources (8)