We investigate the influence of electron-phonon interactions on the dynamical properties of a quantum-dot-cavity QED system. We show that non-Markovian effects in the phonon reservoir lead to strong changes in the dynamics, arising from photon-assisted dephasing processes, not present in Markovian treatments. A pronounced consequence is the emergence of a phonon induced spectral asymmetry when detuning the cavity from the quantum-dot resonance. The asymmetry can only be explained when considering the polaritonic quasiparticle nature of the quantum-dot-cavity system. Furthermore, a temperature induced reduction of the light-matter coupling strength is found to be relevant in interpreting experimental data, especially in the strong coupling regime.