Enhancing Situation Awareness of Maritime Surveillance Operators using Deep Learning based Abnormal Maritime Behaviour Detection

During recent years, we have seen a growing importance of maritime security to ensure the safety of maritime traffic, territorial protection, and the protection of key infrastructure assets. Maritime Surveillance Command and Control systems offer automated tools for enhancing the situation awareness of surveillance operators to improve their decision-making and detect abnormal illicit maritime behavior. Most automated tools in use currently scale poorly with large amounts of data and data sources which raises issues regarding the generalization across space and time and makes detection more prone to errors that may have fatal consequences. The goal of the thesis is to assess the applicability of deep learning methodologies to enhance situation awareness of surveillance operators. In the first part of the thesis, we present an overview of current methods for the detection of maritime abnormalities and discuss how they address some of the issues found in practical application. We then introduce deep learning architectures for the analysis of maritime trajectories. These architectures are based on Recurrent Neural Networks that model trajectories of variable length sequentially. Abnormal trajectories may be detected based on predictive errors in a Sequence-2-Sequence architecture or reconstructive errors in sequential Variational AutoEncoders. This results in models that can analyze maritime trajectories, detect abnormal trajectories, and be trained in a scalable way using large unlabelled datasets. In the second part of the thesis, we evaluate how deep learning architectures may enhance situation awareness of surveillance operators. On the basis of
manually annotated abnormal trajectories related to a recent collision accident, we make a quantitative comparison of the automated detection of abnormal trajectories. Qualitative comparison of flagged trajectories indicates that deep neural networks of different architectures flag different types of abnormal behavior. Therefore, we suggest ensembles of different deep learning architectures in which each member is designed specifically with the detection of a certain type of abnormal behavior in mind. We investigate how to extract the learned normalcy models through interpretable latent variables, but find the encoded information lacking in describing local behavior and insufficient to be used for man-in-the-loop style detections. We propose a two-step clustering approach to describe local behavior and find that this is superior in describing behavioral differences along the same maritime route and for discovering abnormal behavior outside the main maritime routes.