Communication Bounds for Convolutional Neural Networks

Convolutional neural networks (CNNs) are important in a wide variety of machine learning tasks and applications, so optimizing their performance is essential. Moving words of data between levels of a memory hierarchy or between processors on a network is much more expensive than the cost of arithmetic, so minimizing communication is critical to optimizing performance. In this paper, we present new precise lower bounds on data movement for convolutions in both single-processor and parallel distributed memory models, as well as algorithms that outperform current implementations such as Im2Col. We obtain performance figures using GEMMINI, a machine learning accelerator, where our tiling provides improvements between 13% and 150% over a vendor supplied algorithm.

This talk was given as part of a two part minisymposium featuring openly queer mathematicians, and was co-organized by Spectra. The intention of this minisymposium is to promote the participation of LGBT mathematicians in the SIAM annual meeting. Although there is no official data on the percentage of openly LGBT people in mathematics (and very little data on the percentage of openly LGBT people in the general population), it is very likely that openly LGBT people are underrepresented in the mathematics community.