Abstract
Safety is a critical strategic goal of the U.S. DoT according to its RD&T plan with zero fatalities being a grand challenge. This proposal directly addresses pedestrian safety – critical to automotive safety. We target the design of roadside infrastructure at critical points along roadways where pedestrian accidents are likely, such as intersections, sharp curves or hidden driveways. Many such locations involve blindspots, where the vehicle or any single roadside sensor only has a partial view of its environment owing to occlusions. There is rich prior art on sensing systems for blindspot monitoring ranging from the cameras, LiDAR or custom (usually visual) sensors, many of which rely on multiple sensors placed at different vantage points adding to cost and installation burden. What is lacking is a sensing platform that can be placed at a single vantage point, and yet offers a seamless “through-occlusion” imaging, while operating over long-range (hundreds of meters) and offering high-resolution (sub-centimeter).
In this proposal, we design a high-resolution imaging system despite obstructions using two complementary platforms – a mmWave radar and camera. We specifically focus on single-chip automotive mmWave radars that are widely deployed in cars as collision sensors, yet are extremely compact – merely centimeters across. Our key technical insight is that both the mmWave radar and camera have complementary strengths. While the mmWave radar offers extremely high depth-resolution (centimeter-scale at even hundreds of meters) and through-occlusion imaging, its spatial resolution is extremely poor (several degrees). In contrast, cameras offer poor depth resolution (several meters, especially at long range), but high spatial resolution. Our work explores mechanisms to achieve the best of both these systems on a single combined platform.
Unfortunately, a classical data-driven sensor fusion does not directly apply in our case mainly due to the unique attributes of mmWave radar images. Specifically, mmWave radar outputs experience clutter artifacts that must be carefully eliminated to prevent spurious detected objects. Our preliminary work on mmWave-and-camera based depth sensing at IROS’22 explicitly models and corrects for this effect prior to fusing sensed data. Through the proposed work, we seek to generalize this to generate high-resolution 3D point clouds, including through-occlusions.
Beyond technical research contributions, our objective is to demonstrate end-to-end benefits of our system for existing stakeholders. To this end, we will further fully implement and evaluate the system on commodity mmWave platforms and camera systems with the support of Bosch, who has generously offered $100K of in-kind support (lab spaces, equipment, personnel time). Our designs will be deployed at pedestrian intersections, with our first deployment enabled through active collaboration from the City of Pittsburgh Department of Mobility and Infrastructure, who we are in conversations with as a deployment partner. We are acutely aware that pedestrian safety incidents impact under-served, low-income and the housing displaced at a rate much higher than the general population. To this end, we have engaged with the Pittsburgh Parks Conservancy as an equity partner to identify the needs of the community in/around Mellon Square Park in shaping our first deployment.
Description
Timeline
Strategic Description / RD&T
The proposal addresses multiple key themes under the RD&T Plan. The proposal addresses safety, the top priority area of the US DOT according to the RD&T Plan (page III). Safety is listed as the first priority goal in the RD&T plan (page 5). The human element of the safety goal of the DOT is amplified in the safety grand challenge called out in the RD&T Plan: “Advance a future without transportation-related serious injuries and fatalities” (page 11). The design of data-driven safety systems is especially called out with the plan calling for systems that “Evaluate the safety performance of infrastructure design and develop and promote the use of effective safety countermeasures” – a research objective that our proposed work directly falls under (page 17).
The RD&T plan also calls for pedestrian safety systems through a more careful analysis of human-technology interaction. It specifically seeks research innovation to “Learn how people respond to the roadway environment, including signs, markings, and traffic control devices, emerging vehicle and roadway technology, innovative operational changes, and pedestrian and bicyclist safety” (page 18). The proposal designs roadway technology for pedestrian safety, which addresses this precise objective. The RD&T plan also seeks innovation to “Develop and promote effective methods to assess and address traffic safety risks in rural and underserved communities” (page 19) under data-driven safety systems. This proposal’s objective to design safety platforms that specifically serve low-income and homeless communities addresses this objective. More broadly, the proposal adheres to the “safe systems approach” called for in the RD&T plan, which calls for novel roadway infrastructure for safety, stating that “Human error is to be expected, so road infrastructure and vehicle technology must be designed and operated so that deaths and serious injuries are managed through system safety engineering.” (page 21).
Deployment Plan
The proposed work will involve the following key deployment milestones:
Q1) The design and development of a radar-camera sensing pipeline will be performed. The design will be informed by simulation studies to understand anticipated system performance and optimal configuration.
Q2) The design of an initial prototype of the radar-camera sensing system will be made. Initial tests will be performed to test system performance in a lab-setting. The precise testbed site for the proof-of-concept deployment will be identified and surveyed.
Q3) A detailed lab evaluation of the sensing system will be performed to test range, accuracy and overall performance. A detailed plan will be developed for the proof-of-concept deployment.
Q4) A proof-of-concept single vantage point radar-camera pedestrian sensing system will be deployed at a City of Pittsburgh location. System performance parameters such as accuracy and range will be recorded and reported.
Expected Outcomes/Impacts
The proposal directly addresses the safety of automotive systems, specifically pedestrian safety. The proposed work aims to develop fixed infrastructure for pedestrian accident-prone regions to sense the environment and report hazardous situations even if hidden from direct view. The proposed work will create a new model for developing low-cost, easy-to-deploy infrastructure for pedestrian safety. The proposed work has the potential to develop new research products and intellectual property on through-obstruction sensing systems from a single vantage point. Anticipated outcomes will include the development of research publications, technical reports and patents based on the proposed work. We believe that the proposed work also has the potential to shape regulatory requirements for advanced pedestrian safety systems in accident-prone regions.
The proposed work will be implemented and piloted in Pittsburgh through partnerships with the City and Bosch, who has also offered in-kind support in terms of equipment costs, access to lab infrastructure and personnel time. Pedestrian safety incidents disproportionately impact low-income communities and the homeless in particular. The proposed work has a specific mandate to address pedestrian safety in these contexts. In collaboration with our equity partners, we will identify and pilot the first pilot deployment of our platform to address this goal.
Expected Outputs
The proposed work’s key anticipated outputs include:
- Research Impact: The proposed work will develop a fundamentally new approach to fuse radar and camera data to sense objects of interest occluded from the view of a single vantage point. This has important implications for transportation safety and pedestrian safety systems in particular.
Publications: Research developed through the proposed work will be disseminated through publications at major international venues including wireless, sensing systems, robotics and cyber-physical systems venues.
- Data: There is a significant lack of public open-source datasets that combine mmWave radar output alongside camera visual data. Research data developed through the project will be released through open-source permissive licenses to address this gap.
- Software: Source code and associated software systems will be developed to fuse and combine mmWave and camera data.
- Patent Filings: The proposed work will lead to patent filings to protect the intellectual property on the radar-camera sensing system.
- Student Training: One Ph.D. student will be trained during the course of the research. The student will actively collaborate with Bosch and the City of Pittsburgh.
- Pilots: The proposed work will lead to pilot deployments in a City of Pittsburgh location to demonstrate a proof-of-concept through-obstruction pedestrian sensing system.
TRID
The proposal’s uniqueness stems from addressing safe transportation infrastructure for blindspot sensing using a combination of radar and camera data at a single fixed vantage point. The results of the TRiD search using the project keywords resulted in 11 results which are attached. [1] and [5] deal with vehicle maneuvering rather than safety as a primary concern. [2], [6] and [8] provide a broad survey of automotive sensing technologies. [3] and [5] deal with LiDAR and vision based sensing which cannot sense beyond line-of-sight. [7] and [11] advocate communication between multiple vehicles rather than sensing from a single vantage point for safety sensing and dealing with blindspots, which is a complementary approach to the current proposal. Indeed, the current proposal’s key advantage is that it does not require the presence of sensors at multiple vantage points or multiple cooperating vehicles. [8], [9] and [10] deal with holistic vehicular solutions for cybersecurity and driver assistance which complements this work. In summary, the proposed work demonstrates a unique single-vantage point roadside sensing paradigm for safer driving amidst hazards hidden from direct view.
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| swarun@cmu.edu |
Kumar, Swarun |
CMU |
PI |
Faculty - Untenured, Tenure Track |
Budget
Amount of UTC Funds Awarded
$98000.00
Total Project Budget (from all funding sources)
$200000.00
Documents
| Type |
Name |
Uploaded |
| Data Management Plan |
RIAS-DMP.pdf |
Aug. 17, 2023, 4:55 a.m. |
| Presentation |
Next-Generation mmWave Systems |
March 23, 2024, 12:49 p.m. |
| Presentation |
Pushing the limits of mmWave Sensing |
March 23, 2024, 12:49 p.m. |
| Progress Report |
438_Progress_Report_2024-03-31 |
March 23, 2024, 12:50 p.m. |
| Publication |
RadarHD: Demonstrating Lidar-like Point Clouds from mmWave Radar |
March 23, 2024, 12:56 p.m. |
Match Sources
No match sources!
Partners
| Name |
Type |
| Pittsburgh Parks Conservancy |
Equity Partner Equity Partner |
| City of Pittsburgh Department of Mobility and Infrastructure |
Deployment Partner Deployment Partner |
