Abstract
One of the current challenges for autonomous vehicles (AV) is providing provable safety in urban driving and highway driving. LIDAR provides accurate depth and shape information, but remains quite expensive. In this research we explore the viability of fusing 4D radar images with camera images as an alternative to using LIDAR. Our research aims to establish the data fusion of radar and camera images in autonomous driving as a replacement for data fusion using traditional LIDAR, radar, and camera images. Our current approach demonstrates that a trained radar detector can achieve approximately 3.5 meters of Unidirectional Chamfer Distance (UCD) against ground truth LiDAR data on a dataset recently released by Delft University of Technology (DUT) in the Netherlands, which is 28% better than the state of the art (SOTA).
Our approach integrates the images obtained from deep neural network (DNN) 4D radar detectors with conventional camera RGB images. It introduces a novel pixel positional encoding algorithm inspired by Bartlett’s spectrum estimation technique. This algorithm transforms radar depth maps and RGB images into a unified pixel image subspace facilitating the learning of their potential correspondence. Our method effectively leverages high-resolution camera images to train radar depth map generative models, addressing the limitations of conventional radar detectors in complex vehicular environments while sharpening the radar output. We develop spectrum estimation algorithms tailored for radar depth maps and RGB images, a comprehensive training framework for data-driven generative models, and a camera-radar deployment scheme for AV operation.
Our research plan is as follows:
1) Develop a deeper analysis of the training framework to develop more advanced deep learning-based radar detectors.
2) Scale current results on about 100 seconds of representative AV driving from a DUT dataset to a significantly larger number of frames, including a wider range of training and testing scenarios.
3) Quantify additional performance benefit from real-time (instead of current offline) fusion schemes and deployment frameworks for 4D radars and cameras.
4) Use the existing data to measure the underlying latency in the proposed data fusion of 4D radar and camera images. This will help quantify the capabilities and limitations of our scheme.
5) Conduct a detailed error analysis to understand the distribution of error between the point cloud we generate with our data fusion approach against the ground truth. This will help establish any existing bias in the error we obtain in the UCD.
6) DUT’s 4D radar training data set was generated in the Netherlands. We will investigate the augmentation of these data by data collected in collaboration with our deployment partner Blue Fusion, who will allow us the use of one or more of their high-performance radar. This effort will address one of the important pitfalls of machine learning systems, distribution shift or domain adaptation.
The results will be instrumental in providing a cost-effective and weather-resilient solution to the perception part of the provable safety challenge in L4 and L5 Autonomous Vehicles.
Description
Timeline
Strategic Description / RD&T
Section left blank until USDOT’s new priorities and RD&T strategic goals are available in Spring 2026.
Deployment Plan
Sep 30, 2025: Implementation of refined DNN-based camera-radar sensor fusion framework applied to extended dataset
Dec. 31, 2025: Implementation and validation of real-time online camera-radar sensor fusion scheme
Mar. 31, 2026: Obtain results from error and latency analysis of offline and online camera-radar sensor fusion algorithms
June 30: 2026: Augmentation and validation of Delft dataset with Pittsburgh data using 4D radar from deployment partner Blue Fusion
Expected Outcomes/Impacts
• Creation of a camera-radar fusion method that reduces perception cost for autonomous vehicles while maintaining high performance, thereby extending the availability of safe autonomous driving technology
• Validation of the system in simulation and a broad range of datasets
• Documentation of the results in a form appropriate for hand-off to a partner capable of broader deployment
Expected Outputs
• Creation of a novel Bartlett's spectrum-based algorithm for fusing 4D radar and camera data
• Creation of a radar-only system with depth estimation performance good enough to replace LIDAR
• Creation of new datasets for evaluation of camera-radar fusion-based perception
TRID
There are only a few examples of sensor fusion between 4D radar and camera for depth estimation in the literature. The attached TRID document contains 6 papers that resulted from a search on "4D radar camera fusion". The 1st, 3rd, and 5th references are relevant and comparable to our work in terms of their goal and high-level methodology (4D radar-camera fusion), but they do not use the key insight in our approach, which is to translate both the camera and radar information into a common spectrum-based representation. We believe that this aspects of our approach gives it superior performance, and can include in our work comparison with the approach in these three references. The 2nd reference is radar alone. The 4th reference's goal is visual odometry, i.e., localization, rather than depth estimation/scene interpretation. The 6th reference uses the camera for object classification, rather than for increasing radar depth density.
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| malsakab@andrew.cmu.edu |
Alsakabi, Mohammed |
Carnegie Mellon University |
Other |
Student - PhD |
| jdolan@andrew.cmu.edu |
Dolan, John |
Carnegie Mellon University |
PI |
Faculty - Research/Systems |
Budget
Amount of UTC Funds Awarded
$36622.00
Total Project Budget (from all funding sources)
$73244.00
Documents
Match Sources
No match sources!
Partners
| Name |
Type |
| Blue Fusion |
Deployment Partner Deployment Partner |
| RISS, Carnegie Mellon University |
Partner Partner |
