Principal Investigator

Vijayakumar Bhagavatula

Status

Completed

Start Date

Jan. 1, 2015

End Date

Dec. 31, 2015

Project Type

Research Advanced

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

2015 TSET UTC - National

Visibility

Public

Abstract

An  alarming  number  of  fatalities  occur  every  year  due  to  back over  i.e.  drivers  not  noticing objects  behind  their  vehicles  while  reversing.  Due  to  the  increased size  of  the  vehicles  and  the associated  blind  spots,  the  number  of  back over  fatalities  has  been  on  the  rise.  According  to www.kidsandcars.org, at  least  50  children  are  backed  over  by cars  every  week  in  the  US. The objective  of  the  proposed  research  is  to  develop  automated  object  detections  algorithms  from videos captured using rearview cameras. In particular, we will focus on detecting objects behind the vehicle when it is being reversed. 
We proposed to develop  a robust object detection method based on background subtraction. In ideal  conditions,  background  remains  the  same  between the  time when a vehicle  is  parked  and when that vehicle is restarted, resulting in the complete removal of the background and detection of the  objects  in  the  field  of  view.  However,  in  real  world  conditions,  the  appearance  of  the background can change significantly between those two time instants. For example, a driver may park the car in his/her driveway in the evening and pull the car out the next morning.  To achieve acceptable   object   detection   performance   in   real-world   conditions,   we   propose   to   exploit advanced  models  such  as conditional  random  field  models  to  achieve  tolerance  to  illumination variations. Another  research  task  to  be  undertaken  is  predicting  the  most  likely  position  for  a pedestrian several frames into the future so that the vehicle can avoid a potential collision with pedestrians. To develop and evaluate these methods, we also propose to collect a large database that reflects these real-world challenges.
Successful outcomes of this research will include algorithms that can be implemented in vehicles and  produce  robust,  high-accuracy  object  detections  in  images  and  videos  acquired  from production-quality rear-cameras.    

Description

Introduction and Motivation
An  alarming  number  of  fatalities  occur  every  year  due  to  back over  i.e.  drivers  not  noticing objects behind their vehicles while reversing. Due to the increase in the size of the vehicles and the  associated  blind  spots,  the  number  of  backover  fatalities  has  been  on  the  rise.  According  to www.kidsandcars.org,at  least  50  children  are  backed  over  by  cars  every  week  in  the  US. Further, according  to  the  Bureau  of  Labor  Statistics,  over  70  workers  died  from  backover incidents  in  2011.In  an  attempt  to  reduce  the  number  of  such  incidents,  rearview  cameras are becoming  standard  many  vehicles  (in  US,  44  percent  of  2012  models  came  with  rear  cameras standard, according to the automotive research firm Edmunds) to help the drivers see the objects in the rear blind zone. However, according to www.kidsandcars.org, it was found in an Oregon research  study  that  only  one  in  five  drivers  used  a  rearview  camera  when it was  available. The objective  of  the  proposed  research  is  to  develop  automated  object  detections  algorithms fromvideos captured using rearview cameras. 

Research to DateData 
Collection: The first set of data was collected using an HD video camera mounted on the rear side of a sedan. The camera captures images of resolution 1280 by 720 at a frame rate of 29 fps. Videos were collected at the East Campus Garage at Carnegie Mellon University and at the Crane Village Apartment complex in Pittsburgh at two different times of the day (around 10am in  the  morning  and  around  6pm  in  the  evening)  in  the month  of  September  2014.  At  each location  and  at  each  time,  three  sets  of  videos  were  collected:  (i)  with  clear  background, depicting the parking scenario, (ii) with an object of dimension 23in by 19in by 14in placed 3ft behind the car and (iii) with an adult walking from a distance of 15 ft towards the rear side of the car. Each video is about 5 seconds long; the length was intentionally kept short to closely mimic the  short  parking  and  reversing  times  in  a  real-world  scenario.  Sample  images  are  shown  in Figure 1.

Figure 1: Sample Images Captured using the Camera

Technical  Approach: We  are  given  a  collection  of  images  depicting  a  clear  background  (e.g., captured at the time of parking). Our goal is to detect objects of arbitrary shapes and sizes close to  the  rear  side  of  the  vehicle.  We  use background  subtraction to  address  this  problem.  The fundamental  idea in this  approach  is  that  of  detecting  objects  from  the  difference  between  the current  frame  and  a  reference  frame  called  the  “background  image”. We investigated the following two background subtraction approaches so far. 
1. Background Subtraction using Mixture of Gaussians(MoG): In this strategy, each pixel in the  image  is  modeled  using  a  mixture  of  Gaussians  and  an  online  approximation  is  used  to update  the  model.  The  distributions  are  then  analyzed  to  determine  which  are  most  likely  to result  from  a  background  process.  A  given  pixel  is  classified  as  background  if  the  Gaussian distribution that represents it most effectively is a part of the background model and vice versa. 
2. Background Subtraction using Robust Principal Components Analysis (RPCA):The robust PCA algorithm assumes that a given data matrix M can be expressed as a summation of a low-rank matrix L and a sparse matrix S. If we stack the video frames as columns of a matrix M, then the low-rank component L corresponds to the  stationary background (due to the redundancy of video frames) while the sparse component S captures the foreground noise (objects) in the scene. Thus,  an  accurate  decomposition  of  the  test  image  can  facilitate  detection  of  undesired  objects behind the vehicle (from the noise component S).  

Preliminary Results: Sample results of these two background subtraction algorithms are shown in Figure 2. The foreground objects (the box and the human) are accurately identified by both the techniques.  Both  algorithms run  in  near  real-time promising  runtime,  which is  encouraging  for their usage in a real-world scenario. 
Figure 2: Sample Results obtained using the MoG and RPCA Background Subtraction Algorithms 

Proposed Research
In  order  to  improve  the  accuracy  and  the  speed  of  the  automated  object  detection  methods  for rear-camera images, we propose to investigate the following research topics in FY15: 
1.Collect  a large image/video rear-camera  object  detection database reflecting different locations and weather/lighting conditions. Use mannequins to simulate children standing close to a car. We will make the dataset publicly available to facilitate further research in this topic. 
2. We propose to address the effect of illumination and other factors in detecting the objects. We are particularly interested in the setting where the background is derived from images captured at one time of the day (e.g., in the evening)while the test images are captured in a different time of the day (e.g., in the morning) under different environmental conditions, as this depicts a common situation where  a  vehicle  may  be  parked  in  the  driveway  overnight.We  will  develop  and evaluate object detection methods that are tolerant to illumination variations.
3.   We  have  recently  developed a  method  to robustly  detect clear  path ground  surface  with hidden state  conditional  random  field.  Such  a  learning  algorithm  can  be  used  to  distinguish ground surface  from  other  objects  in  an  image.  We  plan  to  use  this  technique  to  detect  the presence of undesired objects behind the vehicle and exploit techniques to fuse the results from the  learning  algorithm  and  the  background  subtraction  strategy.  We  also  plan  to  extend  this learning  algorithm  to  the  online  setting,  where  the  training  dataset  is  augmented  with  the  road surface images captured at the current location, e.g., during parking. 
4. Ensuring the safety of pedestrians is perhaps the most important aspect of driving. Predicting the intent of a pedestrian and whether he is aware of the presence of an approaching vehicle is therefore  of  utmost  importance.  We  intend  to  develop  a computer vision-based  approach  to tackle this problem.

Timeline

1. Collection  of  a large  image/video rear-camera  object  detection database (December 31, 2014)
2. Development   and   evaluation   of object   detection   methods   that   are   tolerant   to illumination variations (March 31, 2015)
3. Development and evaluation of hidden  state conditional random field for background subtraction (June 30, 2015). 
4. Development and evaluation of a computer vision approach for predicting the intent of a pedestrian.

Strategic Description / RD&T

Deployment Plan

We hope to work with the CMU Cadillac SRX team to deploy the developed algorithms.

Expected Outcomes/Impacts

- Demonstration of robust object detection performance on collected databases using developed methods.
- Real-time demonstration of robust rear-camera object detectors in relevant scenarios such as reversing in a driveway or pulling out of a parking space.
- Metrics of interest will include object detection rate, false alarm rate and number of frames processed per second.

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
kumar@ece.cmu.edu	Bhagavatula, Vijayakumar	ECE	PI	Faculty - Research/Systems
shayokc@andrew.cmu.edu	Chakraborty, Shayok	ECE	Co-PI	Faculty - Research/Systems

Budget

Amount of UTC Funds Awarded

$87371.00

Total Project Budget (from all funding sources)

$87371.00

Documents

Type	Name	Uploaded

Match Sources

No match sources!

Partners

No partners!