Heterodyne photon spectroscopy is used for the study of the viscoelastic properties of the liquid interface by studying light scattered from thermally generated surface fluctuations. A theory of a heterodyne apparatus based on a grating is presented, and the heterodyne condition is given in terms of the parameters of the experimental setup. Particular attention is paid to those conditions of the experiment that can cause systematic errors. An instrument function for the apparatus relating the measured output to the experimental parameters of interest is derived. It is shown that the instrument function is generally not Gaussian but that its functional form can depend 9trongly on the parameters of the experimental setup. In particular, it is shown that significant systematic deviations are expected from previous published analysis of the problem. This is primarily due to the nonlinear relation of the surface wave number to the frequency. An algorithm is suggested to arrive at the frequency and width parameters of the spectrum from a typical noisy experimental data set and applied to data obtained from ethanol surfaces.