PyTorch 2.15 tackles the 'one bad GPU' problem

ML admins will be happy to learn that one bad GPU may no longer mean starting an entire training job from scratch. The latest 2.15 release of PyTorch includes a rewritten backend for Nvidia's Collective Communications Library (NCCL), designed to improve fault diagnosis and recovery when GPUs work together.

This latest feature shows how its creators continue to advance the software from being a tool for researchers to being a cornerstone in production AI work.

The interface to the networking library in PyTorch needed a rewrite – it was drowning in technical debt, said PyTorch core developer Tristan Rice, a Meta software engineer specializing in distributed computing, during a live YouTube livestream on Thursday.

NCCL is pronounced "Nickel" and stands for the Nvidia Collective Communications Library. It provides PyTorch with a set of inter-GPU communication primitives, allowing GPUs to communicate within a single-node or across multi-node clusters, which is vital for running inference on large neural networks and for other …