Project: #498 Certification of Connected and Automated Vehicles for Vulnerable Road Users Progress Report - Reporting Period Ending: April 1, 2025 Principal Investigator: Ding Zhao Status: Active Start Date: July 1, 2024 End Date: June 30, 2025 Research Type: None Grant Type: Research Advanced Grant Program: US DOT BIL, Safety21, 2023 - 2028 (4811) Grant Cycle: Safety21 : 24-25 Progress Report (Last Updated: April 16, 2025, 8:03 a.m.) % Project Completed to Date: None % Grant Award Expended: None % Match Expended & Document: None USDOT Requirements Accomplishments The major goal of this project is to explore evaluation methods for the safety of self-driving cars for vulnerable users. The proposed platform will enable the physical evaluation of autonomous vehicles under traffic scenarios involving vulnerable road users (VRUs). Our work is organized into three main tasks. In Task 1, we focus on developing a simulation environment capable of replaying and evaluating critical VRU scenarios, which will serve as our test cases. Task 2 involves generating and integrating VRU-critical scenarios by applying advanced scenario generation algorithms along with accelerated evaluation methods to the simulation environment. In addition, we aim to build a dataset based on the generated scenarios. Finally, Task 3 aims to use a robot to emulate humans in these scenarios in a controlled physical environment. During the reporting period, we achieved milestones in Tasks 1 and 2. For Task 1, we successfully set up the simulation platform. This platform allows for fine control over road conditions—including precise configurations of road layout, traffic controls, and every entity within the scenario—and integrates seamlessly with our critical scenario representation framework. This integration enables us not only to replay and test the critical scenarios but also to conduct thorough virtual evaluations of autonomous driving algorithms. For Task 2, we proposed a novel scenario generation method that combines data-driven with knowledge-based insights. To ensure the realism of the generated scenarios, we adopted a real-world crash dataset from the NHTSA and developed an algorithm that transforms unstructured crash data into a structured format compatible with our simulation environment. To further prevent the generation of unrealistic scenarios, we incorporated both large language models (LLMs) and vision-language models (VLMs). These models exhibit extensive common knowledge and a deep understanding of traffic norms. Moreover, we introduced an innovative algorithm to enhance the spatial reasoning capabilities of the VLMs, which results in more accurate extraction of spatial information and the relative positions of road users. In summary, our solution accurately translates real-world crash data from the NHTSA dataset into a structured representation that fits within our simulation framework, and it provides the flexibility to analyze and convert other real-world traffic datasets into our format. We plan to consolidate this work into a conference paper. During the next reporting period, our efforts will focus on using the proposed method to build a large-scale dataset of VRU-involved critical scenarios. We will design the humanoid robot platform to replicate these scenarios in the physical world, ensuring that our system is fully validated across both virtual and real-world evaluations. Impacts This work is in preparation to be published at CoRL 2025 in September. Other The result was incorporated into CMU course 24784 Trustworthy AI. 131 students enrolled to this course 2024 Fall. Outcomes New Partners N/A Issues N/A