UCCL is an efficient communication library for GPUs, covering collectives, P2P (e.g., KV cache transfer, RL weight transfer), and EP (e.g., GPU-driven)

Ytk-mp4j is a fast, user-friendly, cross-platform, multi-process, multi-thread collective message passing java library which includes gather, scatter, allgather, reduce-scatter, broadcast, reduce, allreduce communications for distributed machine learning.

Summary of call graphs and data structures of NVIDIA Collective Communication Library (NCCL)

Different Implementations of the AllReduce algorithm used in Distributed Deep Learning in c++

This repository outlines a comprehensive guide for training a distributed deep learning model.

Interactive web visualization for understanding collective communication algorithms (as used in NCCL, RCCL, MPI). Learn how AllReduce, Broadcast, Reduce, AllGather and more work step by step.

Sparse Allreduce

Distributed training of a VGG-11 network on the CIFAR-10 dataset

Summary of call graphs and data structures of collective communication plugin in NVIDIA TensorRT-LLM

Modified Dissemination/Bruck algorithm for commutative reduction operations in MPI