LLaDA-8B-Instruct (Large Language Diffusion with mAsking) is an 8-billion-parameter language model developed by the GSAI-ML group (Renmin University of China), trained entirely from scratch without building on any existing autoregressive model. It employs a masked diffusion architecture: during pre-training, tokens are randomly masked and the model learns to iteratively reconstruct them. After supervised fine-tuning (SFT) on 4.5 million pairs, LLaDA-8B-Instruct exhibits instruction-following capabilities comparable to LLaMA3 8B Instruct, but without reinforcement learning. The model is released under the MIT license and available on Hugging Face.

Sources (2)