Project: #407 Alleviating Traffic Congestion: Developing and Evaluating Novel Multi-Agent Reinforcement Learning Traffic Light Coordination Techniques Progress Report - Reporting Period Ending: March 30, 2023 Principal Investigator: Fei Fang Status: Active Start Date: July 1, 2022 End Date: June 30, 2023 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: 2022 Mobility21 UTC Progress Report (Last Updated: March 26, 2023, 4:48 p.m.) % Project Completed to Date: 50 % Grant Award Expended: 50 % Match Expended & Document: 0 USDOT Requirements Accomplishments The goal of this work is to design multi-agent reinforcement learning (MARL)-based algorithms for traffic signal control (TSC) that would be deployable at scale. In the previous reporting period, we reviewed practical considerations that would need to be incorporated into the algorithms by design for deploying MARL in TSC. In this reporting period, we completed the following tasks, which are essential to our goal. First, we built the simulation environment for traffic which include 36 intersections in the center area of Strongsville, OH, and refined it based on feedback from our collaborators at Econonlite and PathMaster. This task is more challenging than what we expected initially, as we need to build the simulation in a way that matches the real-world data of vehicle counts every minute in each lane at each of the intersections. Second, we trained, evaluated, and compared the performance of three different traffic signaling schemes: (1) the default one currently used by Strongsville; (2) the scheme trained with the state-of-the-art MARL algorithm [1]; (3) a heuristic scheme that greedily chooses a phase. The results show that MARL can indeed lead to improvements over the default scheme, but there is still significant room for improvement. Third, we focused on improving the interpretability of MARL-generated policies and proposed an initial algorithm that adapts the recently proposed MAVIPER algorithm [2] to generate decision-tree policies for TSC. [1] Chen, C., Wei, H., Xu, N., Zheng, G., Yang, M., Xiong, Y., Xu, K., and Li, Z. (2020) Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control. In Proceedings of the 2020 AAAI Conference on Artificial Intelligence (AAAI '20). [2] Milani, S., Zhang, Z., Topin, N., Shi, Z.R., Kamhoua, C., Papalexakis, E.E., and Fang, F. (2022) MAVIPER: Learning Decision Tree Policies for Interpretable Multi-Agent Reinforcement Learning. Proceedings of the 2022 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD '22). Impacts We had a field visit to Strongsville and had an in-depth discussion with city engineers in the Strongsville municipality to understand their current practice. We also spent several hours observing the traffic patterns at the major intersections in Strongsville and learned about the hardware available to collect data and control the traffic lights. We have been having bi-weekly zoom meetings with our collaborators at Econonlite and PathMaster. The main purpose of these discussions is to engage the stakeholders in the development of our MARL-based TSC and to prepare for a field test in the future. Other N/A Outcomes New Partners We have been discussing with stakeholders in the city of Strongsville, Ohio, the possibility of testing our algorithm there. Issues We addressed the previously experienced issue with the PTV platform by switching to SUMO platform for traffic simulation. Now we have successfully built the simulator and are continuing the proposed research.