Turbo-dLLM is an Open Source library from Stanford's Scaling Intelligence Lab (including Azalia Mirhoseini) for distributed training of diffusion language models (Block-Diffusion-LLMs). Its core component is the implementation of Context-Sharded Block Parallelism (CSBP), a novel parallelization strategy that accelerates training on very long contexts. The library also includes typed configuration, data pipelines, checkpointing, and optimized CUDA kernels. According to the authors, CSBP accelerates training of the DFlash2 speculative decoder by 2.48x at 512K and by 7.59x at 1M context length on 8x H100 GPUs.

Sources (2)