NCCL Explained: Multi-GPU Training Communication Guide
Abstract Training large language models is never a task that can be completed on a single GPU. When hundreds or thousands of graphics cards need to synchronize gradients and align model parameters in
Sep 9, 20269 min read1
