Project: #304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety Progress Report - Reporting Period Ending: Sept. 30, 2022 Principal Investigator: Keith Redmill Status: Active Start Date: March 1, 2019 End Date: June 30, 2023 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - The Ohio State University Progress Report (Last Updated: Oct. 3, 2022, 9:09 a.m.) % Project Completed to Date: 94 % Grant Award Expended: 96 % Match Expended & Document: 92 USDOT Requirements Accomplishments This project is exploring several potential applications of image processing, including neural network/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles. Topic 1: Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems Our overall research objective is to integrate planning into vision-based deep reinforcement learning-based automated driving systems. Mapping the image space to the vehicle control space is still an open problem. The more recent end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hard coded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision-based DRL framework to alleviate the shortcomings of both worlds. We finished the first phase of this topic and published our work in IEEE ITS Society journals. We also open-sourced our code base. We are still working on the second phase of this topic: investigating hierarchical deep reinforcement learning for end-to-end vision-based automated driving. End-to-end, black box deep learning models are opaque. For safety-critical applications, interpretability is arguably as important as operational robustness. Against this backdrop, we developed a human-interpretable deep reinforcement learning approach. We are currently leveraging a two-layered hierarchical structure with different temporal resolutions. First, a high-level controller provides long-term interpretable options to the low-level controller. The low-level controller is trained to follow these high-level commands with an intrinsic reward while maintaining operational robustness with an extrinsic reward at a higher frequency. Currently, we are testing this approach in a simulated highway environment and comparing its performance against other state-of-the-art DRL solutions. We finished our experiments to evaluate our collaborative point cloud registration algorithm. We developed this algorithm with a partner from the Technical University of Munich. Our approach is to use object-of-interests to align two different point clouds obtained from two different lidar sensors. With this strategy, we aim to improve point cloud registration performance, which is essential for 3D vision-based driving. We published this work in IEEE Access, an open-access journal. Topic 2: Extracting Traffic Information from Transit Vehicle Video This project aims to develop an automatic tool for counting traffic flow using a monocular camera. We are using data collected from OSU on-campus bus service, OSU CABS, to analyze traffic flow using the proposed automatic tool. This activity is essentially complete. We are finalizing a journal manuscript for submission before the end of 2022. Topic 3: Optical Flow for Automated Vehicle Control This activity is complete. Professional development opportunities for our former postdoctoral researcher, who has now transitioned to a full-time Research Associate position, are provided by improving existing skills, conducting research, presenting research findings to others, and increasing duties and responsibilities, including the supervision of students and preparing research project proposals. Additional Activities: We conducted several new experiments and started to use a new simulation environment to test our Hierarchical DRL driving agent. The new "highway environment" enables us to test high-level decision-making capabilities while evaluating low-level driving actions at the same time. Our preliminary results indicate that the proposed solution learns to make optimal high-level decisions in complex traffic scenarios in a completely end-to-end fashion. We are developing a new adversarial pedestrian agent to test our existing driving algorithms better. These adversarial agents will excite our systems to their limits. We aim to identify new modes of failure with this analysis, which can impact future research directions. We have updated the paper citations. Impacts Topic 1- One journal paper got accepted by IEEE Access. B. L. Žagar, E. Yurtsever, A. Peters, and A. C. Knoll, "Point Cloud Registration With Object-Centric Alignment", IEEE Access, 10(2022), pp. 76586-76595. The project code for Hybrid DRL-based Automated Driving is open source and available at: https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving Topic 2- The experiments have been completed, and a paper draft has been completed. We will submit this manuscript to a suitable IEEE journal before the end of this year. Other- The project code for Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention; is open source and available at: https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction Other The project code for Hybrid DRL-based Automated Driving is open source and available at: https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving Other- The project code for Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention; is open source and available at: https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction Outcomes New Partners Topic 1- Technical University of Munich (Department of Informatics, Neuroscientific Systems Theory Group)- Prof. Alois Knoll, Emec Erçelik (PhD student) Issues Topic 1- Hierarchical decision-making can interfere with learning an optimum low-level driving policy. We are conducting ablation studies to identify the problem further End-to-end driving agents can fail against edge cases. We are investigating these edge failure modes with our new adversarial pedestrian attacks. Topic 2- None.