Project: #304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety Progress Report - Reporting Period Ending: March 30, 2022 Principal Investigator: Keith Redmill Status: Active Start Date: March 1, 2019 End Date: June 30, 2023 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - The Ohio State University Progress Report (Last Updated: March 29, 2022, 8:27 a.m.) % Project Completed to Date: 85 % Grant Award Expended: 83 % Match Expended & Document: 82 USDOT Requirements Accomplishments This project is exploring several potential applications of image processing, including neural network/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles. Topic 1: Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems Our overall research objective is to integrate planning into vision-based deep reinforcement learning based automated driving systems. Mapping the image space to the vehicle control space is still an open problem. The more recent, end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hardcoded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision-based DRL framework to alleviate the shortcomings of both worlds. The second stage of this topic, investigating hierarchical deep reinforcement learning for end-to-end vision-based automated driving is still ongoing. The simulation setup has been finalized using CARLA, an open source driving simulator. Currently, state-of-the-art deep reinforcement learning algorithms are being tested for benchmarking purposes. We are currently working on integrating hierarchical decision making structures. We also conducted additional experiments to revise our previous vision-based driving paper submissions titled "Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis with The rendering" and "Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention." Due to the extensive revisions, both of these journal submissions were accepted by IEEE ITS society journals. Finally, we conducted several preliminary experiments to evaluate our collaborative point cloud registration algorithm. We are developing this algorithm with a partner from the Technical University of Munich. Our approach is to use object-of-interests to align two different point clouds obtained from two different lidar sensors. We aim to improve point cloud registration performance, which is essential for 3D vision-based driving, with this strategy. Topic 2: Extracting Traffic Information from Transit Vehicle Video This project aims to develop an automatic tool for counting traffic flow using a monocular camera. We are using data collected from OSU on-campus bus service, OSU CABS, to analyze traffic flow using the proposed automatic tool. We have completed the experimental evaluation of our vision-based traffic extraction algorithms. Our goal is to develop an automatic tool for counting traffic flow using a monocular camera. We are using data collected from OSU on-campus bus service, OSU CABS, to analyze traffic flow using the proposed automatic tool. The experimental validation of the algorithm has been completed. We utilized 3.5 hours of real video imagery from in-service busses and conducted exhaustive ablation studies of variations in the methodology. Our results indicate that the proposed solution can automatically extract traffic information from transit vehicle videos. We finished preparing a journal manuscript. We intend to submit this manuscript before the end of this year. In addition, we helped Dr. Christoph Mertz from CMU to integrate our algorithm into his "Bus on the Edge" platform. Dr. Mertz conducted additional experiments with our framework. We will investigate further collaboration possibilities in the near future with Dr. Mertz's team. Topic 3: Optical Flow for Automated Vehicle Control This activity is complete. Professional development opportunities for our former postdoctoral researcher, who has now transitioned to a full-time Research Associate position, are provided by improving existing skills, conducting research, presenting research findings to others, and increasing duties and responsibilities including the supervision of students. Additional Activities: Extended our research and revised our journal submissions. Two of our submissions were accepted after extensive revisions. The "Photorealism in Driving Simulations: Blending Generative Adversarial Image Synthesis with The rendering" paper were accepted to be published on IEEE Transactions on Intelligent Transportation Systems, and "Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention" paper was accepted to be published on IEEE Transactions on Intelligent Vehicles. We conducted additional fidelity analysis for the photorealism in driving simulations paper and evaluated the effect of object-of-interest rendering on the overall fidelity of simulation-generated imagery. We showed that our proposed framework could learn how to blend object-of-interests with neural network-generated background imagery. This strategy increases the visual fidelity of driving simulations. We also conducted additional pedestrian crossing intention experiments with our vision-based intention prediction algorithm. We used a new benchmark dataset and showed the prediction horizon of the proposed framework. The additional experiments show that the proposed fusion strategy improves pedestrian crossing intention prediction performance. We have updated the paper citations. Impacts Topic 1- Two journal papers were accepted by IEEE ITS Society journals. Topic 2- The experiments have been completed, and a paper draft has been completed. We will submit this manuscript to a suitable IEEE journal before the end of this year. Our Covid-19 social distancing paper has attracted 91 citations (google scholar). Other https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving The project code for "Predicting Pedestrian Crossing Intention with Feature Fusion and Spatio-Temporal Attention" is open source and available at: https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction Outcomes New Partners Topic 1- Technical University of Munich (Department of Informatics, Neuroscientific Systems Theory Group)- Prof. Alois Knoll, Emec Erçelik (PhD student) Issues Topic 1- Assessing the visual fidelity of generative neural network models is not straightforward. We compared the latent feature distribution of real-world images and generated images to solve this issue. Current point cloud registration methods do not perform well for automatic calibration of multiple lidar setups. We plan to solve this issue by developing a new registration algorithm that focuses on object-of-interests to align point clouds. We are investigating the robustness of end-to-end vision systems against adversarial agents. We are planning to evaluate our algorithms against adversarial pedestrian attacks. Topic 2- We are working on extending the usability of the framework. We plan to improve the docker transition for cross-platform deployability. The final goal is to deploy our system to multiple real-world transit networks with different operators.