Machine Learning and Time Series Analysis for Optimization and Anomaly Detection in Field Service

Hybrid Fiber-Coaxial (HFC) networks are one of the most popular infrastructures used to provide cabled internet connection to paying customers. Despite the technology having been around for many years, continuing developments and the low cost of deployment means that it is believed to be an important part of digital infrastructure for many years to come. This significant role, however, comes with challenges, as HFC networks are complex and susceptible to a long range of failures that can happen in different locations in the network. This requires Internet Service Providers (ISPs) to have a large fleet of technicians maintaining the network during daily operation. This is costly in terms of both financial and natural resources, making the optimization of fault detection and localization highly valuable.

Currently, network monitoring is characterized by manual operations, again highlighting the need for automatic and remote approaches. Additionally, many ISPs do not have access to a fully known and updated network topology (the path of connection from a customer modem to the terminal) which is important for accurate troubleshooting and optimal route planning. This makes an algorithm that can reconstruct these topologies using remotely gathered customer data highly valuable.

In this work, we propose two different approaches for remotely detecting anomalies or failures in HFC networks, along with one approach to reconstructing the missing topology. Each of the methods will be using time series data from the network of TDC NET, which is the biggest provider of digital infrastructure in Denmark. One of the biggest challenges with this data is the absence of an accurate ground truth. Our two anomaly detection approaches tackle this problem in two different ways. One approach utilizes the knowledge of a single and wellknown type of error known as Common Path Distortion (CPD). In cooperation with domain experts, we develop a manually labeled dataset where the labels correspond to the presence, respectively, absence of CPD. We evaluate multiple supervised Machine Learning (ML) methods toward their ability to model the dataset and show promising results. We analyze the parameter importance and use that to propose a simple method achieving similar performance while being easier to interpret and implement.

A different approach makes use of a Normalizing Flow (NF), which is a recent generative model, to model the distribution of time series behavior in an unsupervised way. This enables us to detect anomalous behavior in low-density regions of the distribution. Our framework is based on using the NF to estimate the density of a latent representation of the time series computed using an AutoEncoder (AE). While showing promising results, we additionally propose an algorithm for excluding abnormal points from the learned distribution and thus learning only the systematic behavior. We show that the abnormal points are not only pushed out of the distribution but also arrange themselves in clusters that can subsequently be used to identify potential underlying root causes.

Lastly, we evaluate the feasibility of accurately reconstructing the missing topologies using time series data from multiple customer modems. Specifically, we train an encoder that can extract relevant events in time series that are informative with respect to which modems are correlated and which are not. We base our method on an old method from biology used to infer phylogenetic trees from gene sequences. We make multiple contributions to this problem including an updated version of the problem of reconstructing these trees to make it applicable in our case, along with an algorithm for finding the optimal solution. We further contribute by demonstrating the feasibility of embedding this optimization problem directly into a deep learning loss function to learn the informative events for said algorithm and thereby reconstruct the missing topologies.