Principal Investigator

Hae Young Noh

Status

Completed

Start Date

Jan. 1, 2016

End Date

Dec. 31, 2016

Project Type

Research Advanced

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

2016 TSET UTC

Visibility

Public

Abstract

Autonomous vehicles holds tremendous promise in the near future. However, driver intervention
will be needed in emergency or difficult situations. Therefore, real-time monitoring of the driver’s
states, such as attention level, fatigue, and stress, is important to determine safety for transferring
of control. In the past, many single-point on-body sensors and camera systems have been
proposed, but these approaches are often limited to certain environments or require intrusive
sensors on drivers that are difficult to deploy in reality.
In this project, we will use inertial sensors embedded in the vehicle seat for recognizing
driver’s physiological states (including movement, cardiovascular functions) and higher level
states (including stress, fatigue, and attention level). Advantages of the inertial sensor based
driver monitoring system come from its simple and non-intrusive nature, which makes deployment
easy. With previous support from the University Transportation Center, we have developed a
sensing platform consisting of a network of small vibrational sensors distributed inside car seats
and seat belts that can successfully measure posture and breathing of drivers in noisy scenarios.
Building on this hardware, we will continue our effort to develop data processing methods to
extract detailed heart rate, heart rhythm, and movement of drivers. We will then use these
vibrations to infer higher level human states, such as stress level, restlessness, and fatigue. The
main challenge resides in high noise level due to the moving vehicle and sensing constraints
relying only on contacts. To address these challenges, we plan to utilize multi-sourced, highresolution
and high frequency data with hybrid modeling approach to minimize uncertainties in
signal processing and obtain reliable information. For this purpose, we will incorporate physical
models of human body and vehicle dynamics in addition to data-driven models. We will continue
working with our industry partner (Renault) to evaluate our system for laboratory and field testing.    

Description

1. Problem Statement: Autonomous vehicles will define the automotive industry in the near
future. Autonomous vehicles are expected to improve the space utilization of the road systems by
eliminating inefficiencies due to human driving (e.g., large distances between cars to allow for
slow human reaction, parking needs after commute, accidents due to driver distraction, etc.),
while providing extra free time to the drivers [1-3]. The current state of the art involves the use of
externally and internally mounted sensors, such as laser range finders, cameras, inertial sensors,
infrared sensors, etc., to provide the autonomous car with rich view of the world around it. With
these sensors the car can drive autonomously under fairly regular and perfect conditions.
However, the autonomous car would have to give back control to a capable driver when it
is confronted by unusual road or weather conditions (e.g., snow covered roads with invisible road
lane markings, other aggressive road users, unexpected events such as road lane closures, etc.).
Such conditions may interfere with, or even blind, the embedded on-board sensors. Whereas
human drivers have the ability to compensate and adapt to such conditions, the autonomous
vehicle would be limited to only what its sensors can perceive.

Before giving control back to the driver, it is essential for the car to know/estimate the
state of the driver and determine whether the driver is capable of taking control or the car needs
to take other cautious actions. For example, handing back control to a driver who was sleeping,
startled and overly stressed about the situation, or even an absentee driver that was away from
the driver's seat, moving around in the cabin of the car, would be dangerous. By monitoring the
state of the driver through her/his movements and other physiological variables, we can avoid
such situations.

Prior work has explored on-body sensors to maintain attention level of the driver [4-7].
These works often have sensing requirements that require direct contact with the driver, making
them unsuitable for casual drivers. Another approach utilizes camera based systems that monitor
the driver [5, 7]. These systems are often sensitive to different lighting and line-of-sight limitations.
Furthermore, these works focuses on maintaining the driver’s attention, as oppose to understand
the level of inattention due to the current driver state.

With previous support from the University Transportation Center (UTC), we have
developed a sensing platform consisting of a network of small vibrational sensors distributed
inside car seats and seat belts to monitor the driver. Figure 1 shows the backside of a seat
cushion with the sensing platform mounted. Advantages of the inertial sensor based driver
monitoring system come from its simple and non-intrusive nature, which makes deployment
possible. Based on the developed platform, we were able to successfully measure posture and
breathing of drivers in noisy scenarios.

Building on this work, we will continue our effort to develop data analysis methods to 1)
extract detailed driver’s physiological states (including movement, cardiovascular functions) and
2) infer higher level states (including stress, fatigue, and attention level), under various driving
scenarios. The main challenge resides in high noise level due to the moving vehicle and sensing
constraints relying only on contacts. To address these challenges, we plan to utilize signal
processing for multi-sourced, high-resolution and high frequency data with hybrid modeling
approach to minimize uncertainties and obtain reliable information.


2. Proposed Approach: The proposed research consists of three thrusts: 1) Separating the
target vibrations induced by the driver from other noise using multi-resolution signal analysis; 2)
Combining signal- and physics-based models to simultaneously estimate driver’s movement,
heart rate, and heart rhythm from inertial sensor data; and 3) Inferring stress and fatigue level
from driver’s physiological states using machine learning. More details for each thrust is provided
below.

Signal separation: The goal is to decompose vibration signals collected from inertial sensors
into signal components each of which are induced by different sources, such as the driver’s gross
movement, heartbeat, vehicle vibration, etc. The main challenge is that the sensed signal is a
complex mixture of multiple signal components, in which the target signal component to extract
out is often a weak signal with small amplitude overwhelmed by other noise. Further, this mixture
characteristics change over time depending on the driver’s state, which makes the separation
difficult. To overcome the challenge, we propose to use wavelet, a multi-resolution analysis
approach and correlation of multiple sensors to obtain the spatial distribution of the signal timefrequency
spectrum. Based on the spectrum over spatial-temporal and frequency domains, we
can identify signal components that are periodic (e.g., heartbeat), sporadic (e.g., posture change,
speed bumps), and constant (e.g., ambient noise). Our prior work has shown that wavelet was
particularly useful for detecting events of small amplitude from noisy signals [8].

Movement and heart beat extraction: This thrust will incorporate physical models of human
body to data-driven models to extract driver’s movement and heart beat information. Based on
the physical model of human movement, we will investigate the correlation between data from
spatially distributed sensor nodes to estimate the movement. In our previous study, we
demonstrated detecting the presence and posture of the driver whilst in the driver seat using
time-frequency spectrum (Figure 2) and spatial distribution of signal energy (Figure 3),
respectively. Building on this method, we will utilize singularity in wavelet spectrum to detect
changes in the signal due to driver’s movement, since the singularity represents large change in
signal characteristics. The posture change before and after the movement will be inferred from
signal energy distribution and the physiology of human body.
Similar to motion, heartbeat of a person exert a vibration on the car seat. Unlike body
motion, this vibration is small and periodic. Thus, we will identify a set of frequency domain
features from the expected hear rate range from each axis of the accelerometer located near the
driver’s heart in the seat back. We extract features from each axis so as to capture vibrations due
to the person regardless of the exact location and orientation in the chair which they occur. The
heart rhythm will be estimated by reconstructing the signal with detected frequency components.
Often, however, the noise of the car and body motion will overwhelm the heartbeat signal.
Therefore, the system must be able to model the motion noise as well as the heartbeat. A second
observation from our prior modeling approach is that the sensed signal from other noise (e.g., car,
body motion, etc.) will result in higher amplitude “outlier” sensor values, compared to the sensed
signal during no significant noise. By modeling the distribution of the “outliers” and eventually
selectively eliminating a majority of these higher/extreme segments that experience high motion
noise, the system can reduce the effect of high noise level in the accelerometer signal. We will
model the “outlier” as extreme value distribution [9] and perform an outlier detection to remove
noise.

Stress and fatigue inference: Heart rate could be used for cardiac monitoring inside the vehicle
to measure stress level, particularly, the variation of heart rate [10]. By determining heart beats
and model the power spectrum of the heart-rate variability using machine learning, we can then
infer stress. We will improve heartbeat detection accuracy to determine this variability.
In addition to emotional fatigue, physical fatigue plays a large role in determining
attention ability. By utilizing sensors placed near the leg, we will infer the fatigue level of the driver
especially when the muscles are tensed. Furthermore, the system can infer response time and
position of legs for driving style analytics. We will take this work and utilize the model of the noise
developed in the previous thrust to separate muscle vibrations and extract fatigue level.

3. Evaluation: We will evaluate our system in a realistic situation during driving scenarios. In
particular we will work on two main experimentation scenarios: simulated driving and in vehicle
real-world driving. Together with Renault we will utilize their driving facility to test driving stress,
noise model and noise separation algorithms efficiency in a controlled and safe environment. In
addition, we will utilize the research Nissan Leaf vehicle at the lab to test our algorithm with the
subject in the passenger seat on isolated roads in the NASA research center where CMU SV
campus is located. This will provide a more realistic noise environment to validate the algorithms
we will develop.

Timeline

Jan. 1 - Mar. 31, 2016
Develop noise removal and signal separation model for driver’s physiological status induced
signals. Laboratory experiments will be conducted to evaluate the performance of the algorithm.
Mar. 1 - Jun. 30, 2016
Develop and evaluate algorithms for in-vehicle movement and heartbeat recognition. Various
models of human body will be incorporated into the system and evaluated.
Jul. 1 - Sep. 31, 2016
Develop and evaluate algorithms for in-vehicle finer-grain driver state recognition, including
fatigue, stress, and attention level. The system will include algorithms that separate the driving
noise of the car and road, and tested for varying driving scenarios.
Oct. 1 - Dec. 31, 2016
Deployment and validation in various driving scenarios using field experiment as well as the
simulator with our industry partner (Renault). Conduct uncertainty and sensitivity analysis with
collected data.

Strategic Description / RD&T

Deployment Plan

We are developing our algorithms based on the hardware outcome of the previous year’s
research, the system of a network of accelerometers distributed inside car seats and seat belts.
Initially, we will embed and test our algorithms in the car seat in our laboratory settings. This will
serve as initial data collection and iterative algorithm development purposes. As the project
progresses, Renault research (co-located with Nissan research silicon valley and is within a 5
minute drive of the CMU silicon valley campus) will provide access to test our algorithm with their
driving simulator located in their research facility as well as test-drive with a Nissan leaf for more
realistic and on-road testing.

Expected Outcomes/Impacts

The outcome of our research will include a sensing system to infer driver’s physiological states in
a car setting based on vibration sensors. In addition to the hardware developed from previous
year, we will deliver embedded software algorithms to determine and classify the features of the
person’s movement and cardiovascular activities while removing noise and then infer the driver’s
stress and fatigue levels. This will help the autonomous car to make a decision on whether the
driver can take the control back from the car.
We will evaluate the performance of the in-car driver monitoring algorithm in both
simulated and real-world driving scenarios. The driving scenarios will include various situations
such as low speed city driving, stop and go driving, idle, speed bump, high speed freeway driving,
etc. The evaluation metrics will include movement and cardiovascular activity detection accuracy,
false positive and false negative rates.

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
noh@cmu.edu	Noh, Hae Young	CEE	PI	Faculty - Untenured, Tenure Track
peizhang@cmu.edu	Zhang, Pei	ECE	Co-PI	Faculty - Untenured, Tenure Track

Budget

Amount of UTC Funds Awarded

$79983.00

Total Project Budget (from all funding sources)

$79983.00

Documents

Type	Name	Uploaded
Final Report	69_-_TSET_Final_Report_-_Non-intrusive_Driver_Fatigue_and_Stress_Monitoring_Using_Ambient_Vibration_Sensing.pdf	July 27, 2018, 4:18 a.m.

Match Sources

No match sources!

Partners

No partners!