Principal Investigator

Jianbo Shi

Status

Completed

Start Date

Jan. 1, 2013

End Date

Dec. 31, 2014

Project Type

Research Advanced

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

TSET - University of Pennsylvania

Visibility

Public

Abstract

Understanding driver behavior is an essential component in human-centric driver systems. Particularly, driver's interaction with the environment is an important factor in controlling the vehicle, though there have been very few research studies on analyzing driver behavior. Multi-camera array system has a variety of applications because of its improved resolution, frame rate, depth of field, dynamic range and disparity map from such system. In this report we present an implementation of multi-camera array system with GoPro cameras to interact with the external environment of a moving vehicle on streets. So far, our major contribution contains specic analysis of GoPro hardware and protocol, integrating the system with various sensors to collect both internal and external environment information. We also introduce a calibration and rectication method with bundle adjustment for the multi-camera array and optimize the calibration algorithm of First Person Vision glasses. Our goal is to implement realtime intension prediction of drivers with the multi-camera array system.    

Description

1.1 Background
Human behavior analysis based on computer vision is a challenging but valuable research field with lots of
promising applications, such as image understanding, intelligent environment system, interaction between
human and computer, and so on. Generally, human behavior can be analyzed focused on different levels
of the human body such as full body level[8], lower body[7], hand[9], head[4] and foot[6]. For the behavior
analysis of drivers on a vehicle, both the information of the driver’s behavior and the external circumstances
are important. Considering the complexity of external environment, we need a system with high performance
to percept it. Multi-camera array system can meet the requirements quite well.
Multi-camera system is synchronized camera array in certain geometric arrangement, which can function
in various applications depending on the configuration of the camera. The system could be classified into
single-center-of-projection synthetic camera and multiple-center-of-projection camera based on the geometric
configuration[11]. Potential applications include but not limited to improvement of resolution, signal-to-noise
ratio, depth of field and reconstruction of occluded environment. On the other hand, with the advent of low
cost, high resolution compact cameras with synchronization capability such as GoPro cameras[10], building
a multi-camera array system with higher resolution and frame rate, lower cost and complexity becomes
available. Under the circumstances, GoPro cameras become an intellectual choice to prototype such multicamera
array system with reasonable cost and performance [5]. The potential benefit from the system and
the feasibility of the construction become the motivation of the project.

1.2 Purpose
In this study, we want to develop a vision-based framework for driver behavior analysis. By collecting
relevant human driver data from instrumented test vehicle, we want to analyze and develop driver intention
algorithms to predict and estimate the next behavior of the driver.

Timeline

-

Strategic Description / RD&T

Deployment Plan

4 Future Work

We are now regularly collecting data with the GoPro camera array on vehicle. We will focus on relationship between the driver’s intension and behavior. We will also analyze and develop driver intention algorithms with these data and test with different drivers in the following year.
The future work of the multi-camera calibration toolbox is to improve compatibility and stability of various types of camera array and add user-friendly interface. We are trying to figure out the optimal rectification method to move camera in small range for best disparity precision. We would like to publicly release the toolbox in near future.

Expected Outcomes/Impacts

-

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
jshi@seas.upenn.edu	Shi, Jianbo	University of Pennsylvania	PI	Faculty - Tenured

Budget

Amount of UTC Funds Awarded

$0.00

Total Project Budget (from all funding sources)

$

Documents

Type	Name	Uploaded
Final Report	Multi-Camera_System_Based_Driver_Behavior.pdf	March 21, 2018, 8:20 a.m.

Match Sources

No match sources!

Partners

No partners!