Benchmarking machine learning models for collision risk prediction in low-earth orbit

Collision risk estimation and mitigation are essential routine tasks in space operations to keep satellites safe from hazards from other orbiting resident space objects. Timely and accurate space surveillance data are essential for predicting the collision risk and mitigating close encounters in orbit. However, current space surveillance capabilities don’t provide persistent observations for all space objects, and the uncertainty in states, physical properties of space objects, and unknowns in the space environment lead to hundreds of conjunction warning messages weekly, which turn out to be low-risk close to the close approach time. Therefore, the operational cost for analyzing conjunction messages is high currently, and with the advent of mega-constellations and new space surveillance systems that will increase the number of trackable resident objects drastically make it necessary to automate and improve the collision risk prediction.

State-of-the-art machine learning models in the literature are regression, anomaly detection, and time-series models. Although recent research on collision risk prediction using machine learning indicates that there is growing interest in the area, few efforts have been made to investigate machine learning models for resident space object collision risk prediction that are robust to the real world. Therefore, this research not only benchmarks the state-of-the-art machine learning methods based on point estimations but also Bayesian models that can output the uncertainties regarding associated predictions and similarity-based machine learning models that can leverage few-shot learning. The metric used for the benchmarking models is the baseline solution (naive solution) that uses the latest risk value as the predicted collision risk value.

Regression models are used to predict the quantity by regressing the continuous target values. Collision risk prediction using regression models is intended to predict risk values that range from 1e-30 to 1 (note that the lowest risk is 1e-2 in the dataset). The regression models that have been investigated are gradient boosting trees, deep neural networks, and bayesian neural networks. Anomaly detection is concerned with the identification of relatively rare events. Anomalies are different in distribution compared to normal data points. The collision risk prediction for space objects has few anomalies, which are high-risk events. Although the number of objects in space is increasing due to new launches and break-up events, it is still rare that two space objects pass dangerously close to each other. Therefore, this work also casts the collision risk prediction using machine learning as an anomaly detection problem. Gradient boosting trees and similarity-based neural networks have been investigated in the literature for collision risk prediction as anomaly detection. In addition, this work extends the effort to build Bayesian neural networks to be able to address epistemic and aleatoric uncertainties in the data.

Regression using non-Bayesian neural networks show promising results when they are compared to the baseline solution. In addition, similarity-based neural networks without comprehensive hyperparameter search outperform the baseline solution. Further optimization is being conducted to the models to get the largest improvement on top of the best solution reported in the literature, which is the baseline solution that all models are compared.