Unknown · od 2025-02-14 · 2× · naposledy 30. 6. 2026
2
Momentum
LLaDA (Large Language Diffusion with mAsking) is an open-source language model with 8 billion parameters, based entirely on the diffusion principle and trained from scratch without autoregressive modeling. It was developed by researchers from Renmin University of China and Ant Group. The model uses a forward masking process and a reverse generation process via a Transformer that predicts masked tokens. LLaDA was pre-trained on 2.3 trillion tokens and subsequently fine-tuned using Supervised Fine-Tuning (SFT) on 4.5 million pairs.
Vývoj momenta
19.05.17.08.
Vlastnosti
| Benchmark Score (MMLU/Similar) | MMLU 5-shot: 65.9 (LLaDA 8B Base) — surpasses LLaMA3 8B Base (65.4) at equal training tokens (2.3T); GSM8K: 70.7; Math: 27.3; HumanEval: 33.5 |
| Context Window | Native up to 8,192 tokens (8k); LongBench tests show 8k is the supported context window – evaluated at 4k and 8k, content beyond that is truncated |
| Model Size (Parameters) | 8 billion parameters (8B), trained from scratch |
| Price Tier | Free / Open Source (Apache 2.0 licensed, weights publicly available on Hugging Face under GSAI-ML/LLaDA-8B-Base and GSAI-ML/LLaDA-8B-Instruct) |