Project: #304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety Progress Report - Reporting Period Ending: March 31, 2021 Principal Investigator: Keith Redmill Status: Active Start Date: March 1, 2019 End Date: June 30, 2022 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - The Ohio State University Progress Report (Last Updated: April 3, 2021, 12:57 p.m.) % Project Completed to Date: 75 % Grant Award Expended: 75 % Match Expended & Document: 60 USDOT Requirements Accomplishments This project is exploring several potential applications of image processing, including neural network/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles. Topic 1: Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems Our overall research objective is to integrate planning into vision-based deep reinforcement learning based automated driving systems. Mapping the image space to the vehicle control space is still an open problem. The more recent, end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hardcoded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision-based DRL framework to alleviate the shortcomings of both worlds. A novel hybrid approach for integrating path-planning into model-free DRL frameworks was proposed. A proof-of-concept implementation and experiments in a virtual environment showed that the proposed method is capable of learning to drive. We have expanded this work to include hierarchical decision-making capabilities. Currently, we are employing hierarchical deep reinforcement learning algorithms for this task. A paper was published and presented at the IV2020 conference, and the source code of the project is open-sourced under MIT license: https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving Topic 2: Extracting Traffic Information from Transit Vehicle Video This project aims to develop an automatic tool for counting traffic flow using a monocular camera. We are using data collected from OSU on-campus bus service, OSU CABS, to analyze traffic flow using the proposed automatic tool. We finished developing the vehicle-counting software. The current version can count vehicles with respect to travel direction using the footage captured from the OSU CABS buses. In addition, the system can distinguish parked vehicles from vehicles in operation in certain operational domains. During the next reporting period, we plan to publish a conference or journal paper and open-source the code. Topic 3: Optical Flow for Automated Vehicle Control This activity is complete. Additional Activities: In addition to the three official sub-projects, the faculty and students from multiple Mobility'21 projects at OSU along with the postdoctoral research from this project collaborated on four other areas related to image processing. First, a vision-based social distancing and critical density detection system for COVID-19, which has been submitted to the IEEE Transactions on Emerging Topics in Computational Intelligence. Second, a study of the potential for blending generative deep NN adversarial image synthesis with traditional computer graphic rendering to generate photorealistic video images for use in virtual environment simulations, which has been submitted to the IEEE Transactions on Intelligent Transportation Systems. Third, development of a method to predict pedestrian crossing intention in real-time to support safe urban driving. Recent works have shown the potential of using vision-based deep neural network models for this task. However, these models are not robust and certain issues still need to be resolved. First, the global spatio-temporal context that accounts for the interaction between the target pedestrian and the scene has not been properly utilized. Second, the optimum strategy for fusing different sensor data has not been thoroughly investigated. This work addresses the above limitations by introducing a novel neural network architecture to fuse inherently different spatio-temporal features for pedestrian crossing intention prediction. We fuse different phenomena such as sequences of RGB imagery, semantic segmentation masks, and ego- vehicle speed in an optimum way using attention mechanisms and a stack of recurrent neural networks. The optimum architecture was obtained through exhaustive ablation and comparison studies. Extensive comparative experiments on the JAAD pedestrian action prediction benchmark demonstrate the effectiveness of the proposed method, where state-of-the-art performance was achieved. Our code is open-source and publicly available: https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction. Fourth, development of a combined vision and LiDAR point cloud approach to detecting pedestrians and other faraway objects. Learned pointcloud representations do not generalize well with an increase in distance to the sensor. For example, at a range greater than 60 meters, the sparsity of lidar pointclouds reaches a point where even humans cannot discern object shapes. However, this distance should is not very far for fast-moving vehicles; a vehicle can traverse 60 meters under two seconds while moving at 70 mph. For safe and robust driving automation, acute 3D object detection at these ranges is indispensable. We introduce faraway-frustum: a novel fusion strategy for detecting faraway objects. The main strategy is to depend solely on the 2D vision for recognizing object class, as object shape does not change drastically with an increase in depth, and use pointcloud data for object localization in the 3D space for faraway objects. For closer objects, we use learned pointcloud representations instead, following state-of-the-art practices. This strategy alleviates the main shortcoming of object detection with learned pointcloud representations. Experiments on the KITTI dataset demonstrate that our method outperforms state-of-the-art by a considerable margin for faraway object detection in bird’s-eye-view and 3D. Our code is open-source and publicly available at https://github.com/dongfang-steven-yang/faraway-frustum. In addition, the student supported by this project also collaborated with other researchers to explore adversarial testing approaches for automated vehicles and driver assistance systems. The scenario-based testing of operational vehicle safety presents a set of principal other vehicle (POV) trajectories that seek to force the subject vehicle (SV) into a certain safety-critical situation. Current scenarios are mostly (i) statistics-driven: inspired by human driver crash data, (ii) deterministic: POV trajectories are predetermined and are independent of SV responses, and (iii) overly simplified: defined over a finite set of actions performed at the abstracted motion planning level. Such scenario-based testing (i) lacks severity guarantees, (ii) has predefined maneuvers making it easy for an SV with intelligent driving policies to game the test, and (iii) is inefficient in producing safety-critical instances with limited and expensive testing effort. We propose a model-driven online feedback control policy for multiple POVs which propagates efficient adversarial trajectories while respecting traffic rules and other concerns formulated as an admissible state-action space. The approach is formulated in an anchor-template hierarchy structure, with the template model planning inducing a theoretical SV capturing guarantee under standard assumptions. The planned adversarial trajectory is then tracked by a lower-level controller applied to the full-system or the anchor model. The effectiveness of the methodology is illustrated through various simulated examples with the SV controlled by either parameterized self-driving policies or human drivers. Professional development opportunities for a postdoctoral researcher are provided by improving existing skills, conducting research, presenting research findings to others, and increasing duties and responsibilities. Impacts Topic 1- one conference paper has been published and presented on this activity. Source code of the project is open-sourced under MIT license. Topic 2- none at present, still in the software and algorithm development process. But a major goal is to be able to use existing transit camera systems as is, rather than mounting specialized equipment. We do expect to publish the completed software under an appropriate license. Topic 3- two conference papers have been submitted on this activity, one has been accepted. Additional Activities: Our work on monitoring and advising pedestrians on safe distancing and groupings during the COVID-19 pandemic has a potential impact as schools, business, and society in general begins to reopen and there is an increasing potential for close individual contact. Our work on detecting faraway objects and our work on identifying pedestrian intentions has the potential to increase the safety and reliability of autonomous vehicle and advanced driver assistance systems. Finally, our work on adversarial testing approaches has the potential to improve the testing of automated vehicles and driver assistance systems by automatically identifying challenging edge cases of the performance envelope. Other https://github.com/Ekim-Yurtsever/Hybrid-DeepRL-Automated-Driving https://github.com/dongfang-steven-yang/faraway-frustum https://github.com/OSU-Haolin/Pedestrian_Crossing_Intention_Prediction Outcomes New Partners none this period Issues Topic 1- Reward shaping is the biggest challenge for deep reinforcement learning. In order to resolve conflicts between vision-based image-to-control mappings and path planning constraints, conflicting reward terms need to be derived. Topic 2- Each bus has a slightly different camera angle. In addition, image resolution is low. This lowers vehicle detection performance, which in turn lowers the counting performance. Finally, the localization of the bus is necessary for understanding the road segment identification for traffic analysis.