Abstract
The autonomous systems industry in the Pittsburgh region supports 14,900 jobs and $ 1.2 billion in total labor income. It is estimated that within five years, the industry’s total scale will reach $ 10 billion. The key powerhouse is the development of connected and autonomous vehicles (CAVs). Albeit this huge opportunity, one hurdle to this transformative change is the concern of safety. Between 2013 and 2020, 31 states and the District of Columbia enacted legislation related to autonomous vehicles. The impact of state action is starting to manifest through the attraction of efforts that test autonomous systems to regions across the country as companies continue to advance their platforms. Numerous advancements have been developed to mitigate safety risks. For example, simulation tools, closed test grounds, and open corridors have been deployed by companies and universities.
A critical research topic in safety is the evaluation of safety for vulnerable road users (VRUs), such as wheelchair users, people with strollers, vision-impaired people, service-dog users, and e-scooter users. Failure to ensure those people’s safety may result in criticism and backlash from the public and also objection and pushback from regulators. The primary goal of this project is to address this gap by designing and implementing a systematic CAV evaluation certificate program, along with simulation and physical tools, for VRUs.
This objective presents two challenges: (1) the limited data availability; (2) the lack of mature hardware for testing. We plan to address these by leveraging two strengths.
The first strength is our expertise in multi-fidelity Generative AI. To provide stringent assessment, we will leverage our previous work on adversarial, knowledge-based, and data-driven scenario generation to create extensive critical scenarios that pose significant risks to VRUs. PI Zhao has experience in utilizing large language models (LLMs) in autonomous vehicle (AV) legal behavior monitoring. To ensure the coverage of scenarios required by regulations and policies, we will use similar approaches to assist the scenario design. We will also utilize our previous work in accelerated evaluation to boost efficiency. These approaches are intended to mitigate the first challenge.
The second strength is the expertise in both the automotive and robots. We possess the expertise to design systems with both autonomous vehicles and VRUs operated by robots. We will develop a platform that can carry balloon pedestrians/wheelchair users in different terrains and mimic e-scooter users with our wheeled and legged robots. This will offer the advantage of agility and efficiency for self-reconstruction in the event of collisions. Testing robots developed in this project could serve as initial products for a spin-off start-up.
In the next five years, Pittsburgh will encounter increasing competition from regions with signature state and regional initiatives that support autonomy applications. To maintain its position, Pittsburgh must establish programs to reinforce its current innovation ecosystem and root emerging companies and talent in the region. We believe this project will establish a unique strength in the CAV safety evaluation area and secure Pittsburgh’s leading role in the field of autonomy.
Description
Timeline
Strategic Description / RD&T
Autonomous vehicles (AV) and connected autonomous vehicles (CAV) will play an important role in the transportation system. We will follow the “safe design, safety data, safe technology” strategy [RD&T-p 18,19] and develop a comprehensive and automatic AV and CAV evaluation system. We will adhere to the guidance of RD&T, “create an equitable transportation system that provides safe, affordable, accessible, and convenient mobility options for all users” [RD&T-p 11] and “identify and support strategies to increase vulnerable road user safety” [RD&T-p 19], and particular provide evaluation system aiming at the safety of vulnerable road users (VRUs). We will introduce fine-grained categories of VRUs and implement a uniform physical robot platform to mimic the behaviors of various VRUs and enhance the evaluation. We aim to build a transportation system with “zero fatalities” and “advance a future without transportation-related serious injuries and fatalities” [RD&T-p 3,11], and provide “new analytic tools and frameworks to inform and evaluate decisions that support the equitable treatment of all individuals and communities” [RD&T-p33]. In summary, this project aims to address the concerns of the safety of broad categories of VRUs through establishing an efficient and automatic AV and CAV evaluation program, that evaluates AVs and CAVs with controllable multi-fidelity design. The ultimate goal is to ensure the safety of every user of the transportation system.
Deployment Plan
October - November 2024: Simulation Environment Setup and Scenario Structure Setup
October: Initiate the design and configuration of the simulation environment for scenarios with VRUs. Formulate the scenario representation structure and complete the framework that allows scenario setup in the simulation environment. Concurrently, formulate the policies and regulation requirements in the environment and set up the requirements and metrics for evaluation.
November: Implement the interface that allows the user to not only select pre-generated testing scenarios but also control the configuration and create new scenarios.
December 2024 - March 2025: Critical Scenario Generation and Accelerated Evaluation
December: Utilize our previous work in data-driven, knowledge-based, and adversarial scenario generation algorithms [3,4,5,6,7,8] to generate critical scenarios and build a scenario dataset. Integrate LLM to ensure the coverage of scenarios required by polices and regulations [12].
January: Extend the scenario dataset by generating scenarios posing significant risks for VRUs. Enhance the range of scenarios to encompass a broader spectrum of VRUs, by building upon the existing scenarios that involve pedestrians, cyclists, and motorcyclists. Ensure fairness and equity in the testing scenarios by ensuring representation of all categories of VRUs, including but not limited to, people with strollers, e-scooter users, and service dog users, groups for which there is limited data in public datasets.
February: Based on the generated scenarios represented in a structured format, create a corresponding dataset with raw sensor data by leveraging public datasets and generative models. Necessary data will be collected for specific types of VRUs. This will allow the end-to-end evaluation in the simulation and augmented-reality-based physical evaluation.
March: Integrate the generated critical scenarios and scenario generation algorithm in the simulation environment. Integrate LLM for scenario selection and creation. Finish the simulation establishment. Incorporate our previous work on accelerated evaluation algorithms [9,10,11] into the simulation environment to support scenario generation and selection.
April - June 2025: Intelligent Robot Setup and Uniform VRU Platform Setup
April: Based on the scenario dataset and scenario generation dataset, summarize the VRU categories for evaluation. Implement algorithms and programs that enable the wheeled and legged robot to mimic the behavior of VRUs.
May: Complete the hardware modification and testing. Build a uniform platform and interface for the robots that control the movement and collect evaluation results using multiple sensors. Test the robot with the necessary dummies. Prepare the robot for physical testing.
June: Connect the robot with the scenario simulator and implement the program for the robot to enhance the evaluation efficiency. Upon selection of the scenario, this program will allow the robot to maneuver and execute the scenario and self-reconstruct to re-create the scenario after testing.
July - September 2025: Deployment with RIDC
July: Establish the augmented-reality-based evaluation system. Test the evaluation algorithm and program on our autonomous vehicle.
August: Implement the necessary setup in the proving ground for the physical evaluation. Install necessary markers and equipment that allow the robot to precisely construct and reconstruct the scenario. Test the entire system and ensure the support for CAV evaluation.
September: Deploy the evaluation system in the real world and conduct final calibration and adjustment according to feedback from real CAV evaluation cases.
TECHNICAL DESCRIPTION
Motivation:
As autonomous vehicles (AVs) and connected autonomous vehicles (CAVs) continue to evolve with extensive research and development, the evaluation of safety has become a critical topic. According to the 2023 global status report on road safety, only around 30% of death cases were reported for protected road users in 4-wheeled vehicles, while vulnerable road users (VRUs) accounted for the majority of death reports [13]. Therefore, AVs and CAVs must be evaluated comprehensively in scenarios that pose significant risks to VRUs.
The simulation-based and physical evaluations of AV and CAV have been extensively studied. While AV usually can perform safely in normal cases, the performance under critical scenarios still poses uncertainty. Additionally, there has been limited work on evaluations for the safety of VRUs in critical scenarios, and the categories of VRUs are usually limited. When AV companies adopt data-driven self-driving algorithms, there is another level of uncertainty and risk introduced by the limited data availability of minor groups in VRUs. Most works about VRUs focus on pedestrians, cyclists, motorcyclists, and other unprotected mobility users, while minor groups of VRUs who have different behaviors and are under higher risks tend to be classified into these categories. For example, if a vision-impaired person with a service dog is classified as a person with a pet dog, the algorithm may predict the person to have similar behavior as a healthy person. If a child is playing in a parking lot, the algorithm may predict the child as a pedestrian and not expect the child to suddenly run into the route planned by the parking algorithm, as most pedestrians in the dataset are adults or children with guardians and they have more predictable behaviors. Due to the lack of data, there is additional uncertainty when AVs operate in scenarios where minor groups of VRUs are involved, which increases the risks of injuries. To mitigate the risk of injuries, these scenarios must be included in the evaluation. Additionally, improvement and refinement of both software and hardware should be made if AVs and CAVs can cause significant injuries.
To mitigate the risk of harming all categories of VRUs, there is an urgent need for comprehensively evaluating the AV and CAV in critical scenarios that pose risks to all categories of VRUs, especially for VRUs that have different behaviors from the majority groups.
We plan to enhance the strength of the AV and CAV community specifically for the safety of VRUs by establishing a systematic program to efficiently and comprehensively evaluate the AV and CAV. The specific research and development objectives encompass the following components:
1. Scenario generation with the assistance of LLM and accelerated evaluation algorithms:
This project aims to establish a scenario dataset and a uniform framework for critical scenario generation [3,4,5,6,7,8] with the support of accelerated evaluation [9,10,11]. Inspired by the recent achievement of LLM, this project will integrate LLM into scenario generation to ensure the coverage of scenarios required by policies and regulations [13]. Additionally, incorporating LLM will allow users to create customized scenarios efficiently.
2. Uniform platform of robots to automatically execute testing scenarios:
To provide efficient and automatic evaluation, this project aims to build an automatic and uniform platform for intelligent robots. This will allow the wheeled and legged robots to carry dummies and mimic behaviors of various categories of VRUs according to the scenarios selected or created by users. The overarching aim is for these robots to automatically construct the scenario in the proving ground and gather the sensor data for the analysis of evaluation results.
3. Simulation environment and the interface for scenario generation, execution, and evaluation result collection:
To ensure the safety and efficiency of the evaluation process, it is essential to establish a framework that seamlessly connects the scenario generation, scenario execution, evaluation execution, and evaluation result collection. We aim to provide a user-friendly multi-fidelity evaluation system.
Detailed Deployment Plan
Task 1: Simulation Environment Setup
Develop a simulation environment for CAV evaluation. Incorporate scenario design and creation to the simulation environment with structured scenario representation. Implement the metric according to policies and regulations with the assistance of LLM. Implement an interface for users to control the scenario selection and generation with both structured configuration and LLM-based dialogue conversation. Implement an interface for robots to precisely maneuver and execute the scenarios.
Task 2: Critical Scenario Generation
By leveraging our previous work in scenario generation [3,4,5,6,7,8], create a scenario dataset, and implement a uniform framework that allows the simulator and users to access scenarios and scenario generation algorithms to create customized testing scenarios. Generate diver scenarios for more categories of VRUs by leveraging the scenarios involving pedestrians, cyclists, and motorcyclists. Integrate LLM into the scenario generation process to create standard scenarios required by policies and regulations [13]. Additionally, use LLM to enable controllable and customized scenario generation. Integrate accelerated evaluation algorithms from our previous work to support scenario selection and improve the evaluation efficiency.
Task 3: Uniform Testing Hardware for VRUs by Wheeled and Legged Robots
Establish a uniform platform for a broad range of VRUs. Implement algorithms and programs that allow the robot to mimic the behavior of VRUs. Equip the robots with sensors to collect evaluation results. Connect the robot with the simulator and implement 1) the program that controls the robots to execute the scenario generated and selected in the simulator and 2) the program that gathers the sensor data from the robot, analyzes the data and summarizes the evaluation results.
Task 4: Real-world Deployment of the Evaluation System
Implement the physical evaluation system. By combining the simulation environment, scenario dataset and scenario generation algorithms, robots, and the necessary equipment used in the physical evaluation, we will establish an autonomous evaluation system, which allows the user to select the scenario for testing and then automatically execute the evaluation and collect evaluation results in the real world.
Task 5: Document, Reporting, and Refinement according to Feedback
Compile a comprehensive report for the detailed design and implementation of all the components in this project, including the simulator, scenario dataset, scenario generation and selection, and physical robots. Additionally, we will include the challenges and valuable insights, knowledge, and expertise gleaned from the project. Share conclusions, experiences, and actionable recommendations with the autonomous vehicle community.
[1] FOREFRONT: Securing Pittsburgh’s Break-out Position in Autonomous Mobile Systems, 2021.
[2] Ensuring American Leadership in Automated Vehicle Technologies: Automated Vehicles 4.0 , US DOT, 2020.
[3] Ding, Wenhao, Baiming Chen, Minjun Xu, and Ding Zhao. "Learning to collide: An adaptive safety-critical scenarios generating method." In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2243-2250. IEEE, 2020.
[4] Ding, Wenhao, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, and Ding Zhao. "A survey on safety-critical driving scenario generation—A methodological perspective." IEEE Transactions on Intelligent Transportation Systems (2023).
[5] Ding, Wenhao, Baiming Chen, Bo Li, Kim Ji Eun, and Ding Zhao. "Multimodal safety-critical scenarios generation for decision-making algorithms evaluation." IEEE Robotics and Automation Letters 6, no. 2 (2021): 1551-1558.
[6] Ding, Wenhao, Mengdi Xu, and Ding Zhao. "Cmts: A conditional multiple trajectory synthesizer for generating safety-critical driving scenarios." In 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 4314-4321. IEEE, 2020.
[7] Ding, Wenhao, Haohong Lin, Bo Li, and Ding Zhao. "CausalAF: causal autoregressive flow for safety-critical driving scenario generation." In Conference on Robot Learning, pp. 812-823. PMLR, 2023.
[8] Ding, Wenhao, Haohong Lin, Bo Li, Kim Ji Eun, and Ding Zhao. "Semantically adversarial driving scenario generation with explicit knowledge integration." arXiv preprint arXiv:2106.04066 (2021).
[9] Arief, Mansur, Peter Glynn, and Ding Zhao. "An accelerated approach to safely and efficiently test pre-production autonomous vehicles on public streets." In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 2006-2011. IEEE, 2018.
[10] Arief, Mansur, Zhiyuan Huang, Guru Koushik Senthil Kumar, Yuanlu Bai, Shengyi He, Wenhao Ding, Henry Lam, and Ding Zhao. "Deep probabilistic accelerated evaluation: A robust certifiable rare-event simulation methodology for black-box safety-critical systems." In International Conference on Artificial Intelligence and Statistics, pp. 595-603. PMLR, 2021.
[11] Chen, Rui, Mansur Arief, Weiyang Zhang, and Ding Zhao. "How to evaluate proving grounds for self-driving? A quantitative approach." IEEE Transactions on Intelligent Transportation Systems 22, no. 9 (2020): 5737-5748.
[12] Hong Wang, Wenhao Yu, Chengxiang Zhao, Jiaxin Liu, Xiaohan Ma, Yingkai Yang, Jun Li, Weida Wang, Xiaosong Hu, and Ding Zhao, “Online Legal Driving Behavior Monitoring for Self-driving Vehicles", Nature Communications, 2023
[13] World Health Organization. 2023. "Global Status Report on Road Safety 2023." World Health Organization. https://www.who.int/publications/i/item/9789240086517.
Expected Outcomes/Impacts
The primary anticipated outcome of this project is an automatic AV and CAV physical evaluation system that aims at the safety of VRUs. The system contains a simulation environment for users to create and select critical scenarios for testing, intelligent robots to automatically execute the scenario set up, and a system that connects the simulator, robots, and sensors to collect the evaluation results. We will collaborate with PennSTART to design and deploy the real-world evaluation system. Additionally, we aim to introduce augmented reality into the evaluation, establishing an evaluation system that supports not only simulation-based and physical evaluation but also an efficient evaluation paradigm that can achieve better tradeoff between efficiency and sim-to-real gap. PI Zhao will be the general chair of the IEEE International Automated Vehicle Validation Conference 2024. We will integrate the outcome of this project into the conference and call for works that will contribute to this research topic and community.
Expected Outputs
- An AV and CAV evaluation system that can automatically create critical scenarios posing risks to VRUs based on users’ requirements.
- A scenario database and a uniform scenario generation framework to provide extensive critical scenarios for most categories of VRUs.
- A uniform robot platform to automatically construct and reconstruct the scenarios and collect sensor data about the evaluation results.
- Potential patent filing and tech transfer.
TRID
According to the TRIS database, this project would mark the first endeavor on connected autonomous vehicles (CAVs) physical evaluation programs that focus specifically on the safety of vulnerable road users (VRUs). This project will complement the existing project “Vehicle-in-Virtual-Environment (VVE) Method for Developing and Evaluating VRU Safety of Connected and Autonomous Driving”. The difference is that instead of coarse-grained categories of VRUs, namely pedestrians, cyclists, and motorists, our project will include more fine-grained groups, especially for the minor groups and those who have different behaviors. Another difference is that instead of virtual environments, our project aims to develop both the software and hardware components required by the physical evaluation program. This project will also complement the existing project “Safety Assessment of the Interaction Between the Autonomous Shuttle Bus and Vulnerable Road Users”. The difference is that our project aims to develop an evaluation system for general AVs and CAVs instead of the shuttle bus. Additionally, we will develop a uniform physical evaluation program instead of modeling the interaction between the vehicle and VRUs.
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| dingzhao@cmu.edu |
Zhao, Ding |
Carnegie Mellon University |
PI |
Faculty - Untenured, Tenure Track |
Budget
Amount of UTC Funds Awarded
$95000.00
Total Project Budget (from all funding sources)
$100000.00
Documents
Match Sources
No match sources!
Partners
| Name |
Type |
| RIDC |
Deployment Partner Deployment Partner |
| City of Pittsburgh |
Equity Partner Equity Partner |
