This paper demonstrates how significant improvement in frequency response and directivity of a loudspeaker may be obtained by optimizing the local properties of the materials for the diaphragm and surround. Performance is investigated as the considered frequency range and off-axis requirements are progressively expanded. The results are generated by optimizing the values and layout of stiffness, mass, and damping of both the speaker diaphragm and surround. This is accomplished using a density and gradient-based optimization technique in conjunction with a fully coupled finite element model of the loudspeaker and the surrounding acoustic domain. The targeted frequency range is from 600 Hz up to 10 kHz and the range for the directivity is from 0° to 30°. The results show that a completely flat on-axis response is achievable even for very broad frequency ranges and that a reasonably flat response over a wide directivity can be obtained as well. The results presented in this research assume that complete design and production freedom are available.