Korel: Mitigating Stragglers via Real-Time Automatic Mixed Precision in Distributed Deep Learning Environments

In distributed deep learning systems, straggler nodes are a primary factor in delaying gradient synchronization during synchronous training, thereby diminishing overall efficiency. This issue is particularly pronounced in multi-tenant environments, where resource contention from concurrent tasks exacerbates node slowdowns. Existing approaches typically mitigate this problem by excluding slower nodes or reducing their influence during gradient aggregation, which often leads to resource underutilization. In this study, we introduce a novel solution that leverages Automatic Mixed Precision (AMP) to tackle the straggler problem. Our method employs a MAD-based threshold to detect workers impaired by resource contention and dynamically applies AMP to accelerate their computations. Once contention subsides and node performance recovers, AMP is deactivated to maximize efficiency. Experimental evaluations across diverse deep learning tasks reveal that our approach significantly reduces time-to-accuracy in synchronous Distributed Data Parallel (DDP) environments, enabling rapid convergence to high accuracy. In addition, by effectively counteracting the delay imposed by stragglers-quantified by our mitigation metric which indicates that up to 79.4% of the straggler-induced delay is offset-our approach achieves convergence up to 14 % faster than traditional BSP methods under adverse conditions.