Project: #304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety Progress Report - Reporting Period Ending: Sept. 30, 2020 Principal Investigator: Keith Redmill Status: Active Start Date: March 1, 2019 End Date: June 30, 2022 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - The Ohio State University Progress Report (Last Updated: Oct. 5, 2020, 6:28 a.m.) % Project Completed to Date: 62 % Grant Award Expended: 52 % Match Expended & Document: 60 USDOT Requirements Accomplishments This project is exploring several potential applications of image processing, including neural network/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles. Topic 1: Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems Our overall research objective is to integrate planning into vision-based deep reinforcement learning based automated driving systems. Mapping the image space to the vehicle control space is still an open problem. The more recent, end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hard-coded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision based DRL framework to alleviate the shortcomings of both worlds. During this period, a novel hybrid approach for integrating path planning into model-free DRL frameworks was proposed. A proof-of-concept implementation and experiments in a virtual environment showed that the proposed method is capable of learning to drive. The proposed integration strategy is not limited to path planning. Potentially, the same state-space modification and reward strategy can be applied for integrating vehicle control and trajectory planning modules into model-free DRL agents. Finally, the current implementation was limited to output only discretized actions. A paper describing this work was accepted for the IV 2020 Symposium. In that paper, we trained a vision-a* hybrid from scratch and we showed that our method learned to drive faster than a pure vision-based approach. Future work will focus on enabling continuous control, extending to other planning and control tasks, and real-world testing. Topic 2: Extracting Traffic Information from Transit Vehicle Video This project aims to develop an automatic tool for counting traffic flow using a monocular camera. In addition, we propose to utilize data collected from the OSU on-campus bus service, OSU CABS, to analyze traffic flow over time and space using the proposed automatic tool. Currently, the software is in the development stage. The current version can count vehicles with respect to travel direction using the footage captured from the OSU CABS buses. However, the system has trouble distinguishing parked vehicles from vehicles in operation. This is a problem because parked vehicles should not be counted as traffic participants. We intend to write and publish a conference/journal paper and open-sourcing the code. Topic 3: Optical Flow for Automated Vehicle Control During this period, we analyzed a small dataset of real (on-road) front camera images in different weather conditions with the proposed pipeline. Without further adjustments to the optical flow recognition, we showed that we can accurately predict the control actions of the vehicle in above 70% of the cases in cloudy weather, and above 60% in rainy weather. It would be expected that these numbers would improve when tuning the optical flow field to discard motion from rain droplets in the windshield (for rainy weather) and objects outside of the scope of the road lanes (in general). This analysis was added to the final submission of the IV2020 paper. Professional development opportunities for a postdoctoral researcher are provided by improving existing skills, conducting research, presenting research findings to others, and increasing duties and responsibilities. Additional Activities: In addition to the three official sub-projects, the faculty and students from multiple Mobility'21 projects at OSU along with the postdoctoral research from this project collaborated on two other areas related to image processing. First, a vision-based social distancing and critical density detection system for COVID-19, which has been submitted to the IEEE Transactions of Industrial Informatics for publication. Second, a study of the potential for blending generative NN adversarial image synthesis with traditional computer graphic rendering for use in virtual environment simulations, which has been submitted to the 2021 AAAI conference on Artificial Intelligence for publication. Impacts During this period: Topic 1- one conference paper has been published on this activity. Source code of the project is open-sourced under MIT license. Topic 2- none at present, still in the software and algorithm development process. But a major goal is to be able to use existing transit camera systems as is, rather than mounting specialized equipment. We do expect to publish the completed software under an appropriate license. Topic 3- two conference papers have been submitted on this activity, one has been accepted. Additional Activities: Our work on monitoring and advising pedestrians on safe distancing and groupings during the COVID-19 pandemic has a potential impact as schools, business, and society in general begins to reopen and there is an increasing potential for close individual contact. Other none Outcomes New Partners Topic 1- none Topic 2- Christoph Mertz, Principal Project Scientist (CMU), and his student Justin Chian, masters student (CMU Robotics Institute) developed a parked vehicle detection tool. We used this tool and their experience as a starting point for the vehicle counting project, though much of our processing work has now developed in a different direction. Topic 3- none Issues Topic 1- Reward shaping is the biggest challenge for deep reinforcement learning. In order to resolve conflicts between vision-based image-to-control mappings and path planning constraints, conflicting reward terms need to be derived. Topic 2- Each bus has a different camera angle. In addition, image resolution is low, and some cameras may have a dirty or wet lens. Finally, of these conditions can change over time. This lowers vehicle detection performance, which in turn lowers the counting performance and effects the project of detected vehicles on ground/road coordinates. Successful operation is a goal of the project, but it does great problems. Finally, the localization of the bus is necessary for understanding the current road segment identification for traffic analysis, and this data is not recorded in a manner that makes it directly accessible for our processing software.