Static, Scalable, and Dynamic Compression Strategies for Efficient Neural Speech Processing

Audio and speech processing systems enable a wide range of applications, such as telecommunications, remote collaboration, and assistive technologies, and are often a crucial component of audio-centric products and devices. In many cases, these applications mandate strict latency, privacy, and connectivity constraints, thereby forcing processing to occur on-device, where computational resources are further limited by battery life. Compared to traditional signal processing techniques, deep learning provides substantial performance improvements in speech processing tasks; however, realizing these gains on resource-constrained devices is still challenging.
In practice, deploying such systems to embedded targets involves training a smaller network or applying compression techniques; yet, since task performance is directly related to computational cost, any design choice will inevitably bring a negative impact to one of these two critical aspects. Moreover, by focusing solely on worst-case operating conditions, a systematic inefficiency arises: under favorable scenarios, such as low background noise, models are overprovisioned, performing more computations than necessary; conversely, in challenging acoustic environments, the trade-off may result in suboptimal performance. Even after discounting these external factors, modern devices often feature multiple, simultaneous, and variable workloads, which pose additional limitations on the amount of available computational resources and make large, static models even less amenable to edge execution.
This latter issue can be mitigated by scalable models, which expose multiple operating points, each characterized by a different cost and, in turn, performance; this is equivalent to deploying multiple models of different sizes without suffering the additional storage burden. While certainly useful, these models still require external control to determine how resources should be allocated.
Dynamic neural networks extend this paradigm by adapting their computational structure to the input, in full autonomy. Intuitively, yet somewhat tangentially, this approach bears some similarity to a popular model of human cognition, whereby the brain defaults to simple heuristics when performing familiar tasks, only resorting to more effortful processes when a situation demands it. More concretely, dynamic techniques induce sparsity in the network (e.g., along its depth or width) so that only a subset of its operations is executed for a given input, thereby enabling an efficient utilization of computational resources. While this class of models has received substantial attention in the computer vision community, its application to the audio domain is a more recent endeavor, and the lack of foundational work addressing the peculiarities of audio signals and tasks in the context of dynamic neural networks may pose a barrier to widespread adoption.
In this thesis, we tackle the challenge of deploying deep learning models for speech processing on resource-constrained devices, focusing on tasks such as speech enhancement, keyword spotting, and speech quality prediction, which are ubiquitous and fundamental aspects of most audio products. We do so from the perspective of network design and training, and through three complementary axes: compression, scalability, and dynamism. Specifically, we introduce early exiting — a form of depth-wise dynamism — to a recurrent neural network for speech enhancement, documenting its behavior across different training strategies and suggesting architectural adaptations to mitigate the performance degradation caused by the additional exit points. We subsequently develop a modular mixed-precision quantization scheme that selectively reduces the bit width of multiply-accumulate operations within the model, as well as a binarization-aware training procedure resulting in a model that does not require multiplications at all. The sparsity induced by this latter technique would later serve as inspiration for a fully dynamic network for speech enhancement that performs structured pruning at runtime, based on the input; in another work, we turn our attention to the learned pruning decision, discover that they are remarkably informative, and leverage them as input features for a number of ancillary tasks such as gender or noise classification. Lastly, motivated by the need for a more tractable and structured form of sparsity, we investigate the application of slimming (i.e., a way of converting a static network into a scalable one by treating discrete subsets of its parameters as independent submodels) in a fully dynamic network, proposing a method for end-to-end training that accounts for the non-differentiability of the discrete decisions.
Experimental results demonstrate that the proposed compression strategies could yield substantial reductions in model size and computational cost, potentially motivating further research on tailored hardware accelerators. On the other hand, by learning to allocate computation adaptively, dynamic neural networks achieve input-dependent efficiency gains; moreover, the learned dynamic policies exhibit interpretable behavior, providing insights into how different parts of the model contribute to the overall task and what aspects of the input are deemed relevant. These findings confirm the potential of dynamic and scalable neural networks as viable solutions for efficient speech processing, while highlighting the need for better hardware support and more representative evaluation protocols to fully realize and assess their potential.