Abstract:Distributed deep learning has become one of the most popular research fields in the entire deep learning.Increasingly large amounts of data,models and algorithms will rely on the highly intelligent idea of divide and conquer to process.However,the distributed deep learning system has encountered many new challenges in the rapid development,the most important of which is the effective transmission of large amounts of data in the limited network bandwidth of the distributed environment,in order to make full use of computing resources to efficiently train high precision deep neural networks.In response to this problem,this article discusses different techniques of data transmission compression from three perspectives,including parameter synchronization,model aggregation,and gradient compression,which are respectively used for interleaving data transmission,reducing the frequency of data exchange,and the amount of data transmitted each time.The future development trend of distributed deep learning data transmission compression mechanism is discussed and prospected.