A Min-Heap-Based Accelerator for Deterministic On-the-Fly Pruning in Neural Networks

This paper addresses the design of an area and energy efficient hardware accelerator that supports on-the-fly pruning in neural networks. In a layer of N neurons, the accelerator selects the top K neurons in every timestep. As K is fixed, the runtime of the pruned network is deterministic, which is an important property in real-time systems such as hearing aids. As a first contribution, we propose to use a min-heap for the top K selection due to its efficient data structure and low time complexity. As a second contribution, we design and implement a hardware accelerator for dynamic pruning that is based on the min-heap algorithm. The heap memory storing the top K neurons and their index is realized as a 3-port standard cell-based memory implemented with latches. As a third contribution, we evaluate the energy savings from pruning of a gated recurrent unit used in a neural network for speech enhancement (regression task). Our experiments demonstrate energy savings of ~78% without degrading the SNR improvement, and up to ~93% while reducing the SNR improvement by 0.1 - 1.11 dB. Moreover, the overhead of the hardware accelerator constitutes negligible ~0.5% of the total energy. The accelerator is implemented in a 22nm CMOS process.