Project: #304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety Progress Report - Reporting Period Ending: Sept. 30, 2021 Principal Investigator: Keith Redmill Status: Active Start Date: March 1, 2019 End Date: June 30, 2022 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - The Ohio State University Progress Report (Last Updated: Oct. 6, 2021, 3:06 p.m.) % Project Completed to Date: 81 % Grant Award Expended: 90 % Match Expended & Document: 91 USDOT Requirements Accomplishments This project is exploring several potential applications of image processing, including neural network/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles. Topic 1: Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems Our overall research objective is to integrate planning into vision-based deep reinforcement learning based automated driving systems. Mapping the image space to the vehicle control space is still an open problem. The more recent, end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hardcoded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision-based DRL framework to alleviate the shortcomings of both worlds. The second stage of this topic, investigating hierarchical deep reinforcement learning for end-to-end vision-based automated driving is still ongoing. The simulation setup has been finalized using CARLA, an open source driving simulator. Currently, state-of-the-art deep reinforcement learning algorithms are being tested for benchmarking purposes. We will integrate hierarchical decision making structures after the completion of the benchmarking step. We also conducted a collaborative study with Technical University of Munich (TUM) about developing robust vision-based perception algorithms for automated driving. We introduced a novel multi-frame Lidar-Camera fusion method for detecting vehicles, pedestrians, and cyclists using end-to-end neural networks. This work has been presented and published in the proceedings of 2021 IEEE Intelligent Vehicles Symposium. The codebase is open-sourced and can be accessed via the following link: https://github.com/emecercelik/Temp-Frustum-Net Topic 2: Extracting Traffic Information from Transit Vehicle Video This project aims to develop an automatic tool for counting traffic flow using a monocular camera. We are using data collected from OSU on-campus bus service, OSU CABS, to analyze traffic flow using the proposed automatic tool. The initial experimental validation of the algorithm has been completed, comprising both 3.5 hours of real video imagery from in-service busses and an exhaustive ablation studies of variations in the methodology. The preliminary results indicate that the proposed solution can automatically extract traffic information from transit vehicle videos. A paper draft has been prepared, and it will be submitted to an IEEE journal before the end of the year. A first version of this algorithm, as a docker image, has been provided to Christoph Mertz for possible testing on his "Bus on the Edge" platform. Topic 3: Optical Flow for Automated Vehicle Control This activity is complete. Professional development opportunities for our former postdoctoral researcher, who has now transitioned to a full-time Research Associate position, are provided by improving existing skills, conducting research, presenting research findings to others, and increasing duties and responsibilities including the supervision of students. Additional Activities: Extended and submitted COVID-19 Vision-based distancing paper to the journal Sensors, which was accepted for publication. Extended our research and submitted a revised paper including responses to reviewer comments for our research on blending generative deep NN adversarial image synthesis with traditional computer graphic rendering to generate photorealistic video images for use in virtual environment simulations to the IEEE Transactions on Intelligent Transportation Systems. Attended and presented 4 papers at ITSC (3 related to Mobility21 UTC). We have updated the paper citations. Impacts Topic 1- One conference paper has been published and presented on this activity. Source code of the project is open-sourced. Topic 2- The experiments has been completed, and a paper draft has been prepared. We will submit this manuscript to a suitable IEEE journal after proofreading before the end of this year. Our Covid-19 social distancing paper has attracted 49 citations (google scholar). Other https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving https://github.com/dongfang-steven-yang/faraway-frustum https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction https://github.com/emecercelik/Temp-Frustum-Net Outcomes New Partners Topic 1- Technical University of Munich (Department of Informatics, Neuroscientific Systems Theory Group)- Prof. Alois Knoll, Emec Erçelik (PhD student) Issues Topic 1- Multi-frame feature alignment for vision based end-to-end driving systems are challenging. We are implementing state-of-the-art sequential models such as LSTMs, GRUs, and Transformers to tackle this problem. Topic 2- Thorough experimental validation requires well defined ground truth information. Even though we have real world data collected by in-campus transit busses, the ground truth counts still need to be pre-processed for each segment and pass of the bus.