Improving urban operations through sequential decision-making : integrative policy based learning approaches for logistics and transport operations

Abstract

Urban transport operations are at the heart of city infrastructure and efficiency, playing a pivotal role in the quality of urban life. The real-time aspect of these systems is fundamental to addressing the immediate needs of a city's dynamic environment. This thesis explores the optimisation of real-time urban transport operations through the lens of two interconnected applications: emergency service dispatch during crises and the scheduling of electric buses within public transit systems.

Amidst the backdrop of the COVID-19 pandemic, the dispatch of emergency services has emerged as a critical challenge, requiring rapid and strategic deployment of resources in the face of uncertainty and time-critical conditions. As orders arrive ceaselessly, transport operators are tasked with making swift assignment decisions to meet demands. Agents are matched with customers for tasks of uncertain duration, and they move directly to the next job or return to a depot according to specific applications. We utilise a REINFORCE algorithm to train models that minimise service delay, incorporating spatio-temporal data and a novel 'withhold' option for agents. Our method outperforms benchmark methods, reducing delay and travel time for agents. However, the method's effectiveness diminishes in situations where the system is either too overloaded or underutilised.

The second application focuses on the operation of electric buses, which are increasingly integral to sustainable urban transport operations. We integrate Reinforcement Learning (RL) with Mixed Integer Linear Programming (MILP) to tackle this problem. This hybrid framework takes advantage of RL's ability to learn from the outcomes of state-action pairs and the computational efficiency of MILP, which ensures solution feasibility through its constraints. Our evaluation, conducted on both hypothetical and real bus network data, indicates that our approach surpasses the benchmark optimisation methods, in terms of reducing penalties for missed service trips, and improving the average and variance of headway between buses. The effectiveness of our proposed framework is particularly pronounced in highly stochastic environments.

In order to address the above problems, this research has been structured to achieve three key objectives. The first objective involves developing a versatile simulation environment capable of accommodating any decision-making agent, be it heuristic algorithms or RL. The second objective is about the design of architectures and training algorithms. The proposed framework leverages state-of-the-art approximators to overcome the "curse of dimensionality.", and uses suitable training algorithms to train approximators. The third objective revolves around the effective incorporation of operational constraints into the RL framework via a novel masking scheme. This innovation ensures compliance with real-world logistical parameters and optimises decision-making.

This thesis makes three contributions to the academic literature. First, it advances the field through the development of a novel Reinforcement Learning framework integrated with semi-Markov Decision Processes, designed to facilitate real-time assignment decisions in dynamic environments. Second, the thesis provides valuable insights into the formulation of efficient assignment policies that can withstand the variability of customer demand and the limitations of resources. Third, it contributes methodologically to the optimisation of real-time service network coordination, demonstrating the effectiveness of incorporating operational constraints and strategic decisions.