#304 Image Processing Approaches to Traffic Situation Understanding, Risk Assessment, and Safety

Principal Investigator
Keith Redmill
Start Date
March 1, 2019
End Date
June 30, 2023
Project Type
Research Advanced
Grant Program
FAST Act - Mobility National (2016 - 2022)
Grant Cycle
Mobility21 - The Ohio State University


This project will explore several potential applications of image processing, including NN/deep learning technologies, to the analysis of traffic scenes involving passenger and transit vehicles.  We outline three potential applications below- the exact distribution of effort and topics addressed will depend on the availability of student and research staff.
1.	Investigating the Safety and Robustness of End-to-End Vision-Based Automated Driving Systems
2.	Methodologies for extracting information from transit vehicle video for traffic flow estimation
3.	Optical flow for automated vehicle lane following    
1.	A. Investigating the safety and robustness of end-to-end vision-based automated driving systems
Increasing the mobility of people and goods via automated transportation requires robust and safe intelligent vehicle systems. Computer vision is one field that can create new frontiers towards this end. The exteroceptive capabilities of vision-based scene perception algorithms is approaching human-level performance. However, mapping the image space to the vehicle control space can still be considered an open problem. The more recent, end-to-end Deep Reinforcement Learning (DRL) based automated driving algorithms have shown promising results for mapping image pixels to vehicle control signals. However, pure learning-based approaches lack the hard-coded safety measures of model-based counterparts. We propose a hybrid approach for integrating a model-based path planning pipe into a vision based DRL framework to alleviate the shortcomings of both worlds. Our overall research objectives are to develop novel algorithms to integrate planning into deep reinforcement learning frameworks. We will identify the variables needed for our methodologies and develop, evaluate, and over time improve algorithms to increase the robustness of learning-based autonomous driving systems.

2.	Methodologies for extracting information from transit vehicle video for traffic flow estimation
Transit vehicles cover a nontrivial fraction of the road network as they follow their assigned routes and have the potential to act as traffic probes providing observations distributed over significant periods of time and space.  Many transit systems have installed cameras and video capture systems that, as a byproduct of their primary monitoring and safety functions, also capture traffic scenes and the presence and motion of vehicles, bicycles, and pedestrians surrounding the transit vehicle.  We will explore working with image processing research collaborators at CMU and OSU to share video collected from the OSU TTM CABS campus transit system, identify the variables needed for our traffic flow estimation methodologies, and develop, evaluate, and over time improve algorithms and methodologies to automatically process CABS video data.  These results will be evaluated and compared with results generated using our existing techniques.  Possibilities for expansion into new traffic variables, for example extracting vehicle speeds which are currently extracted from cost prohibitive LIDAR sensors, will be explored.   We expect the developments and applications of this activity to allow us to deliver useful information to a stakeholder (OSU TTM) and to demonstrate the potential for large scale implementation.

3.	Optical flow for automated vehicle lane following
We will examine the application of optical flow with sliding mode control for vehicle navigation and compare traditional methods of optical flow extraction to NN/Deep Learning approaches.  This will initially be done in Carla, a driving simulator with virtual reality environment capabilities, and may later be extended to the Transportation Research Center testing facility.  Topics of interest include optical flow and focus of expansion, road and road boundary potential field design, and gradient tracking sliding mode controller for lateral and longitudinal control.

Strategic Description / RD&T

Deployment Plan

Expected Outcomes/Impacts

Expected Outputs



Individuals Involved

Email Name Affiliation Role Position
hillstrom.7@osu.edu Hillstrom, Stacy The Ohio State University Other Other
redmill.1@osu.edu Redmill, Keith The Ohio State University PI Other
yurtsever.2@osu.edu Yurtsever, Ekim The Ohio State University Co-PI Faculty - Researcher/Post-Doc


Amount of UTC Funds Awarded
Total Project Budget (from all funding sources)


Type Name Uploaded
Data Management Plan dmp-Redmill-2019.docx May 1, 2019, 1:57 p.m.
Project Brief Redmill_et_a_2019_slides.pptx May 10, 2019, 8:07 a.m.
Progress Report 304_Progress_Report_2019-09-30 Oct. 1, 2019, 7:52 a.m.
Publication Visual potential field based control for autonomous navigation in unseen environment March 31, 2020, 2:57 p.m.
Publication OpticalFlow_IV2020_final.pdf April 3, 2021, 12:54 p.m.
Publication Risky_Action_Recognition_In_Lane_Change_Video_Clips_Using_Deep_Spatiotemporal_Networks_With_Segmentation_Mask_Transfers.pdf March 31, 2020, 3:06 p.m.
Progress Report 304_Progress_Report_2020-03-31 March 31, 2020, 3:07 p.m.
Publication Photorealism_in_Driving_Simulations_Blending_Generative_Adversarial_Image_Synthesis_With_Rendering.pdf Oct. 3, 2022, 8:44 a.m.
Publication IV2020_0252_FI3.pdf April 3, 2021, 12:54 p.m.
Publication sensors-21-04608.pdf Oct. 6, 2021, 1:33 p.m.
Progress Report 304_Progress_Report_2020-09-30 Sept. 30, 2020, 4:30 p.m.
Publication Faraway-Frustum_Dealing_with_Lidar_Sparsity_for_3D_Object_Detection_using_Fusion.pdf March 29, 2022, 8:07 a.m.
Publication Predicting_Pedestrian_Crossing_Intention_With_Feature_Fusion_and_Spatio-Temporal_Attention.pdf Oct. 3, 2022, 8:45 a.m.
Publication A_Modeled_Approach_for_Online_Adversarial_Test_of_Operational_Vehicle_Safety.pdf March 29, 2022, 8:10 a.m.
Progress Report 304_Progress_Report_2021-03-31 April 3, 2021, 12:57 p.m.
Publication Temp-Frustum_Net_3D_Object_Detection_with_Temporal_Fusion.pdf March 29, 2022, 8:16 a.m.
Progress Report 304_Progress_Report_2021-09-30 Oct. 6, 2021, 1:36 p.m.
Publication Integrating_Deep_Reinforcement_Learning_with_Model-based_Path_Planners_for_Automated_Driving.pdf March 29, 2022, 8:16 a.m.
Publication Pedestrian_Emergence_Estimation_and_Occlusion-Aware_Risk_Assessment_for_Urban_Autonomous_Driving.pdf March 29, 2022, 8:18 a.m.
Progress Report 304_Progress_Report_2022-03-30 March 29, 2022, 8:20 a.m.
Publication A finite-sampling, operational domain specific, and provably unbiased connected and automated vehicle safety metric May 2, 2022, 9:47 a.m.
Publication A Formal Safety Characterization of Advanced Driver Assist Systems in the Car-Following Regime with Scenario-Sampling May 2, 2022, 9:48 a.m.
Publication A Formal Characterization of Black-Box System Safety Performance with Scenario Sampling May 2, 2022, 9:48 a.m.
Publication Towards Guaranteed Safety Assurance of Automated Driving Systems with Scenario Sampling: An Invariant Set Perspective May 2, 2022, 9:49 a.m.
Publication A modeled approach for online adversarial test of operational vehicle safety May 2, 2022, 9:49 a.m.
Publication Towards Guaranteed Safety Assurance of Automated Driving Systems with Scenario Sampling: An Invariant Set Perspective (Extended Version) May 2, 2022, 9:50 a.m.
Publication Point_Cloud_Registration_with_Object-Centric_Alignment.pdf Oct. 3, 2022, 8:51 a.m.
Progress Report 304_Progress_Report_2022-09-30 Oct. 3, 2022, 9:09 a.m.
Progress Report 304_Progress_Report_2023-03-31 March 31, 2023, 1:30 p.m.
Final Report Mobility21_Project304_ImageProc-FinalReport-tagged.pdf Aug. 3, 2023, 10:02 a.m.
Publication Automated Traffic Surveillance Using Existing Cameras on Transit Buses Oct. 23, 2023, 8:48 a.m.
Progress Report 304_Progress_Report_2023-09-30 Oct. 23, 2023, 8:48 a.m.
Publication Automated Traffic Surveillance Using Existing Cameras on Transit Buses Oct. 23, 2023, 8:48 a.m.

Match Sources

No match sources!


Name Type
The Ohio State University Deployment Partner Deployment Partner
Technical University of Munich Deployment Partner Deployment Partner