The picosecond dynamics of excitons in colloidal CdSe nanorods are directly measured via their 1s to 2p-like internal transitions by ultrabroadband terahertz spectroscopy. Broadened absorption peaks from both the longitudinal and transverse states are observed at 8.5 and 11 THz, respectively. The onset of exciton-LO phonon coupling appears as a bleach in the optical conductivity spectra at the LO phonon energy for times > 1 ps after excitation. Simulations show a suppressed exciton temperature due to thermally excited hole states being rapidly captured onto ligands or unpassivated surface states. The relaxation kinetics are manipulated and the longitudinal transition is quenched by surface ligand exchange with hole capturing pyridine.