End-to-end learning for the nonlinear fiber channel

The ever–growing data traffic demand has been driving the optical networks to constantly evolve over the years. To efficiently meet this demand, the future optical communication systems have to be continuously upgraded to offer higher capacity. In recent years, machine learning has gained traction as an approach to further increase capacity. A promising direction is the use of autoencoders, which can be utilized to optimize both the transmitter and receiver jointly for a specific channel model and
performance metric, also known as end–to–end learning. The methods presented in this thesis contribute to the state–of–the–art in end–to–end learning for optical communication systems with three main contributions. First, optimization of an autoencoder utilizing a gradient–free training method based on Bayesian filtering is studied. Typically, an autoencoder requires a differentiable channel model to be optimized using a gradient–based algorithm. However, this requirement can be limiting since practical channel models often have non–differentiable parts. An autoencoder for geometric constellation shaping is employed to numerically validate the proposed method on four different channel models and compare it to established optimization methods. Second, a robustness optimization procedure to learn a constellation shape robust to varying channel conditions utilizing an autoencoder was proposed. The optimization procedure relies on varying parameters of a simple differentiable channel model, whereas the testing is performed on a more realistic channel. It is demonstrated that a constellation robust to signal-to-noise ratio and laser linewidth uncertainties can be learned. The analysis is expanded by breaking down the signal-to-noise ratio uncertainty into launch power and amplifier noise figure uncertainties. Third, the influence of a limited hardware bit resolution on an autoencoder for geometric constellation shaping is analysed. Two performance metrics are considered, mutual information and generalized mutual information. The results of the analysis demonstrate that generalized mutual information requires a higher bit resolution than mutual information.