Linear and Nonlinear Impairment Compensation in Coherent Optical Transmission with Digital Signal Processing

Digital signal processing (DSP) has become one of the main enabling technologies for the physical layer of coherent optical communication networks. The DSP subsystems are used to implement several functionalities in the digital domain, from synchronization to channel equalization. Flexibility and effectiveness of DSP contribute to reducing costs and increase reliability of optical communications systems. The work presented in this thesis focuses on DSP subsystems for coherent optical communication systems. In particular, the contributions presented in this thesis relate to the following topics: (I) Kerr nonlinearity compensation, (II) spectral shaping, and (III) adaptive equalization. For (I), original contributions are presented to the study of the nonlinearity compensation (NLC) with digital backpropagation (DBP). Numerical and experimental performance investigations are shown for different application scenarios. Concerning (II), it is demonstrated how optical and electrical (digital) pulse shaping can be allied to improve the spectral confinement of a particular class of optical time-division multiplexing (OTDM) signals that can be used as a building block for fast signaling single-carrier transceivers. Finally, regarding (III), original contributions to equalization in coherent optical receivers are proposed, consisting of a new approach to analyzing and design equalizers for systems where receivers or transmitters may be subject to front-end imperfections. Numerical and experimental validations are performed to evaluate the proposed methods. In conclusion, the results presented in this thesis contribute to the state-of-the-art of DSP for coherent optical communication over single-mode fibers (SMFs). The techniques investigated have the potential to improve performance and reliability of such systems, ultimately enabling throughput and transmission reach improvements for the next generations of coherent systems.