The nonlinear evolution of the modulational instability of whistler waves coupled to fast magnetosonic waves (FMS) and to slow magnetosonic waves (SMS) is investigated. Results from direct numerical solutions in two spatial dimensions agree with simplified results from a set of ordinary differential equations obtained from a Hamiltonian formulation of the governing equations. The long-time evolution of the modulational instability for the FMS-coupling shows a quasi-recurrent behavior with a slow spreading of the energy to higher and higher mode numbers. For the SMS-coupling, no recurrent behavior is found and the energy is gradually leaking to higher mode numbers while the spatial evolution of the modulation tends to develop small scale ''spikes.'' (C) 1995 American Institute of Physics.