Bayesian Machine Learning for Simulation Metamodeling

Policymaking is a complex task that demands skilled leaders who can navigate the intricate web of competing interests and priorities, a challenge exacerbated by recent crises such as environmental issues, pandemics, and economic instability.

In transportation, these challenges invoke policy goals such as reducing congestion and achieving carbon neutrality, which introduce numerous variables and uncertainties, often overwhelming decision-makers. Policymakers typically simplify the problem by examining a limited set of scenarios, potentially overlooking the probabilistic nature of the issues and missing valuable insights. Decision-making in transportation often involves simulation models, but they cannot fully handle the computational demands of diverse scenarios. The central issues are the abundance of scenarios and the complexities of simulation models.

These challenges boil down to two critical issues in simulation-based decision-making. First, the multitude of scenarios leads to cognitive overload, constraining decision-makers due to bounded rationality. Second, the substantial computational complexity of detailed models limits the number of options that can be analyzed in a reasonable time frame. Innovative approaches to policymaking and simulation-based studies are essential to tackle the complexities of our ever-changing world. An efficient approach to alleviate the computational challenges is to employ a metamodel (also known as a surrogate model). These simulation metamodels approximate the simulation model’s function with mathematical simplicity, speed, and interpretability. They offer a computationally economical solution to the burdens of simulation-based policy analyses, providing a higher level of insight into complex systems.

This dissertation focuses on mitigating the two aforementioned issues in simulation-based decision-making by advancing the metamodeling techniques. Within the practical context, the dissertation emphasizes enhancing the interpretability and explainability of resultant metamodels to reduce cognitive overload. Methodologically, it underscores the use of Bayesian non-parametric models, with a particular emphasis on Gaussian processes and their integration with Bayesian active learning, encompassed within Bayesian machine learning. These methods are employed to tackle the complexities arising from computational
demands. Therefore, the thesis consists of two parts. In the first part, the emphasis is on advancing techniques to enhance the explainability of metamodels and devising a framework for seamlessly integrating these improvements into existing active learning metamodeling approaches. The findings suggest that active learning metamodels can assist in decision-making by making the exploration of scenarios more manageable, particularly using the explainable sub-component in the form of SHAP values. In the second part, the thesis delves into the design of novel active learning strategies, followed by the creation of
a new model aimed at reducing both the number of required simulations and the training time for the metamodel itself. The results suggest that the extra Bayesian information from the fully Bayesian Gaussian processes and the proposed mixture of Gaussian processes are more data efficient and thus reduce the computational burden further. All these methodologies and frameworks are applied to an air traffic management (ATM) simulator, effectively demonstrating their tangible value in addressing real-world problems.

This thesis concludes with a brief discussion on short- and long-term next steps toward the goal of computationally efficient and explainable data-driven decision-making for simulation-based policymaking.