Abstract
Intersection safety is critical for all traffic participants, but especially for vulnerable road users (VRU) such as pedestrians and cyclists. Recent autonomous driving advances allow mitigation of several human driver risk factors, such as fatigue and recklessness. However, state-of-the-art autonomous driving technology also has limitations. An individual vehicle’s perceptual field-of-view can be compromised by nearby occluding objects, greatly reducing detection accuracy, and distant objects can be difficult to detect. Limitations in detection accuracy introduce further challenges in the downstream tasks of object tracking, trajectory prediction, and motion planning. In this project, we develop techniques for cooperative sensing at intersections to address these challenges and enable more effective identification of potential collisions involving VRUs and then combine them with novel CAV collision-mitigating actions to improve VRU safety.
Cooperative sensing: Building on recent research in cooperative object recognition by multiple connected autonomous vehicles (CAVs) near an intersection [1,2], we will develop extended techniques for tracking and predicting the trajectories of travelers. We will first evaluate different approaches to reconciling shared feature maps (specifically the use of distributed Kalman filtering methods [3,4] versus newer transformer-based approaches [5]) to determine a baseline object tracking procedure. Second, we will consider the tracking performance benefit of either additionally incorporating information from fixed-position camera/lidar sensors at the intersection (e.g., as would be possible at signalized intersections that use such sensors to support adaptive traffic signal control systems) or substituting them for CAV sensing altogether. Finally, we will adapt and apply these results, which traditionally assume vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication and focus on accurate sensing of vehicles, to the sensing of VRUs. We will use available traffic data (e.g. [3]) to evaluate our initial technology results and use our East Liberty Surtrac/PedPal testbed site to test live VRU detection.
Collision mitigation: To enable safe response to projected collisions with sensed VRU, the project will also investigate CAV strategies for taking evasive action. Recently deep reinforcement learning (DRL) has successfully dealt with autonomous driving tasks. We aim to use DRL to train an agent across various intersection scenarios with human-like reactive agents to learn to decode social maneuvers from the past observed trajectories of the surrounding road users and execute a safe strategy for crossing the intersection. The developed algorithms will be tested on CommonRoad scenarios with reactive agents trained on a wide range of human behaviors.
The proposed work will use state-dropout-based curriculum RL [6] with Control Barrier Functions (CBF) to react in cases wherein the ego vehicle comes close to collision with other road users. This allows the ego agent to receive the CBF’s initial help to learn how to react safely, and then as the agent learns, lift the CBF constraints in order to cross intersections faster without compromising on safety. The proposed approach is coupled with existing state-dropout-based curriculum RL where future states of the ego agent are initially available as privileged information to ease learning for the RL agent but are subsequently removed so the RL agent can efficiently cross the intersection.
Description
Timeline
Strategic Description / RD&T
This proposal addresses the “Zero Fatalities” Grand Challenge (Ch. 1, p. 3) and two of its desired outcomes (Ch. 2, p. 16): 1) “People no longer accept a high risk of fatality or serious injury as a cost of mobility.” 2) Vehicle...designs incorporate proven, active and passive safety features that protect vehicle occupants and non-occupants.” It aligns with the US DOT RD&T primary purpose of “Promoting safety” (Ch. 1, p. 5) and with the US DOT strategic goals of “Safety” and “Transformation” (Ch. 1, pp. 5-6). Each of these strategic goals is dealt with separately below.
Safety: The proposal addresses two of this goal’s research priorities: Human Factors & Data-Driven System Safety (Ch. 2, p. 17, Table 3). Under the Human-Technology Interactions heading of Human Factors, it “Explore[s] the effects of new technologies, including automation, on travel behaviors" (Ch. 2, p. 18).Under the Safety Technology heading of Data-Driven System Safety, it “Leverage[s] innovative technologies to monitor, predict, and plan ways to reduce injuries and fatalities among the transportation workforce and traveling public" (Ch. 2, p. 19).
Transformation: The proposal addresses one of this goal’s research priorities: New and Novel Technologies (Ch. 2, p. 50, Table 6). Under the Automation heading of this priority, it “Conduct[s] research to develop an effective and efficient safety assessment framework for automated systems across all modes of transportation” and “Develop[s] best practices for safe interaction of automated roadway vehicles with existing vehicles, … pedestrians, cyclists, and motorcyclists” (Ch. 2, p. 60).
Deployment Plan
During the first quarter of the project work will focus on:
(1) Development and evaluation of techniques for cooperative detection and tracking of travelers moving through the intersection from multiple sensors. Most prior work in this area has focused on cooperation between CAVs approaching an intersection, and we will start with this problem formulation to enable us to contrast the effectiveness of our candidate approaches with prior results obtained on existing, publicly available vehicle traffic data sets. The initial goal will be to understand the strengths and limitations of previous approaches and develop a baseline approach that overcomes these limitations and provides state-of-the-art results. Once we have established this baseline, we will shift attention to (1) adapting this baseline approach to multimodal detection of vulnerable road users, specifically pedestrians and cyclists, and (2) extending the approach to utilizing data multiple sensors that are instead mounted at fixed locations at the intersection, both in isolation and in addition to exploiting sensor data obtained from approaching CAV vehicles. During this phase, we will work with our deployment partner Miovision to ensure that our assumptions are well aligned with those that are anticipated to be important for the Intersection Safety System (ISS) concept that we intend to jointly submit to the DOT Intersection Safety Challenge. We anticipate that one likely path to eventual deployment of our cooperative sensing techniques will be through successful participation in this Challenge.
(2) Development and implementation of intersection collision-avoidance behaviors and initial testing/validation in the CommonRoad simulator. CommonRoad is a simulator created by the TU Munich that has a large number of real-world scenarios and accompanying real-world data in which planning and behaviors methodologies can be comprehensively tested.
During the 2nd quarter of the project work will focus on the following activities:
(1) We will augment our techniques for cooperative tracking of vulnerable road users to include methods for trajectory prediction. We will approach this problem similarly, by understanding previous work in this area, drawing on potential expertise by Miovision in this area and the trajectory prediction needs of our jointly conceived ISS concept, and performing comparative experimental analysis to rationalize our eventual solutions.
(2) Refinement of intersection collision-avoidance behaviors and extensive testing/validation in CommonRoad simulator. Basic testing of collision-avoidance behaviors on a 1/10th scale autonomous car.
(3) Integration of the cooperative sensing and collision avoidance aspects of the project in simulation.
During the 3rd and 4th quarters of the project, we will perform the follow activities:
(1) We will focus on refining the cooperative sensing framework and testing it in the field. We will utilize one of the intersections in our East Liberty testbed that is outfitted with Miovision detection equipment and attempt to augment these sensors with additional lidar sensing capability. We will also generalize our framework to also incorporate real-time information obtained from smartphone apps in the possession of vulnerable road users (such as our PedPal safe intersection crossing app for pedestrians with disabilities, and apps that provide analogous tracking capabilities for other classes of vulnerable users). We will attempt to collect a stream of traffic data and provide benchmark results, to provide a data set and benchmark performance results, and make them available to the research community for future research.
(2) To the extent possible and consistent with safety dependent on the described existing live testbed capabilities, implementation of ADAS-type driver warnings in unsafe intersection situations.
Expected Outcomes/Impacts
We believe our research results will present new opportunities significantly improving the safety of vulnerable road users as they navigate through busy intersections. In the shorter-term, these safety improvements will likely come from configurations of sensors that are mounted at different locations in the intersection together with mobile sensors (such as smartphone apps) that are possessed by vulnerable road users and have real-time traveler-to-infrastructure connectivity. In the longer-term, as connected autonomous vehicles (and their sensing capabilities) become more prevalent on the roadways and they incorporate collision avoidance strategies following from this project, we anticipate significant further improvement in intersection safety.
Expected Outputs
The primary anticipated output of this project is a prototype intersection safety system blending two subsystems with novel capabilities: 1) a perception system fusing information from all available sources including connected vehicles and infrastructure; 2) a collision avoidance system combining flexibility of maneuver with safety guarantees based on based on cutting-edge safe control techniques.
This research will produce new techniques and approaches for real-time cooperative detection of potential collisions that involve vulnerable road users from data obtained from multiple diverse sensing devices that are situated both at the intersection and possessed by various traveler types approaching and moving through the intersection. We also anticipate the creation and dissemination of a new intersection traffic data set to further stimulate future research in this currently fledgling area.
This research will also produce new techniques for mitigating potential collisions that are detected. This will include both autonomous vehicle responses, which will be demonstrated in simulation and on an F1Tenth car, and ADAS or smartphone alarms in live testing.
It is anticipated that these techniques will on the one hand yield new benchmark results that will find their way into publications and push future research in this area forward, and on the other hand, lead to invention disclosures and software artifacts that will promote subsequent deployment and eventually lead to realization of substantial reduction in traffic accidents that involve vulnerable road users.
TRID
There are a few cited references that have developed and or contemplated the design of real-time cooperative perception systems from multiple sensor streams, but with few exceptions this is all very recent (2022 or 2023) and research in this area is just getting under way and exploratory. In our earlier work we in fact built on such a recent research effort (published in 2022) and were able to better solve pieces of their approach and produce new better performance on benchmark data sets. The general area of cooperative sensing is just starting up and it is far from a solved research problem at this point. One novelty of our proposed research over all of these efforts is our emphasis on vulnerable connected road users (like pedestrians and cyclists).
The collision avoidance portion of our proposed work is unusual in providing safety guarantees using Control Barrier Functions (CBF). The nature of the guarantee we aim to provide is probabilistic, because other vehicle and VRU models are necessarily uncertain, but prior work has been largely unable to provide guarantees of any kind.
Prior work can be used to provide datasets for testing and baselines for performance comparison.
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| hsukuanc@andrew.cmu.edu |
Chiu, Hsu-kuang |
Robotics Institute |
Other |
Student - PhD |
| jdolan@andrew.cmu.edu |
Dolan, John |
Robotics Institute |
Co-PI |
Faculty - Research/Systems |
| dkalaria@andrew.cmu.edu |
Kalaria, Dvij |
Robotics Institute |
Other |
Student - Masters |
| sfs@cs.cmu.edu |
Smith, Stephen |
Robotics Institute |
PI |
Faculty - Research/Systems |
Budget
Amount of UTC Funds Awarded
$95000.00
Total Project Budget (from all funding sources)
$200000.00
Documents
Match Sources
No match sources!
Partners
| Name |
Type |
| Miovision Inc. |
Deployment Partner Deployment Partner |
| pathVu |
Deployment & Equity Partner Deployment & Equity Partner |
| PennDOT |
Deployment & Equity Partner Deployment & Equity Partner |
