Geometric Constellation Shaping for Fiber-Optic Channels via End-to-End Learning

Ognjen Jovanovic (Invited author), Francesco Da Ros (Invited author), Darko Zibar (Invited author), Metodi P. Yankov (Invited author)

Research output: Contribution to journalJournal articleResearchpeer-review

38 Downloads (Pure)


End-to-end learning has become a popular method to optimize a constellation shape of a communication system. When the channel model is differentiable, end-to-end learning can be applied with conventional backpropagation algorithm for optimization of the shape. A variety of optimization algorithms have also been developed for end-to-end learning over a non-differentiable channel model. In this paper, we compare a gradient-free optimization method based on the cubature Kalman filter, model-free optimization and backpropagation for end-to-end learning on a fiber-optic channel modeled by the split-step Fourier method. The results indicate that the gradient-free optimization algorithms provide a decent replacement to backpropagation in terms of performance at the expense of computational complexity. Furthermore, the quantization problem of finite bit resolution of the digital-to-analog and analog-to-digital converters is addressed and its impact on geometrically shaped constellations is analysed. Here, the results show that when optimizing a constellation with respect to mutual information, a minimum number of quantization levels is required to achieve shaping gain. For generalized mutual information, the gain is maintained throughout all of the considered quantization levels. Also, the results imply that the autoencoder can adapt the constellation size to the given channel conditions.
Original languageEnglish
Article number10124361
JournalJournal of Lightwave Technology
Issue number12
Pages (from-to)3726-3736
Publication statusPublished - 15 Jan 2023


  • Optimization
  • Quantization (signal)
  • Symbols
  • Channel models
  • Decoding
  • Signal processing algorithms
  • Mutual information


Dive into the research topics of 'Geometric Constellation Shaping for Fiber-Optic Channels via End-to-End Learning'. Together they form a unique fingerprint.

Cite this