#47 Non-Intrusive Driver Distraction Monitoring Using Vehicle Vibration Sensing
- Principal Investigator
- Hae Young Noh
- Status
- Completed
- Start Date
- Jan. 1, 2017
- End Date
- Aug. 31, 2018
- Project Type
- Research Advanced
- Grant Program
- MAP-21 TSET National (2013 - 2018)
- Grant Cycle
- 2017 TSET UTC
- Visibility
- Public
Abstract
Driver distraction is responsible for more than a quarter of the 1.3 million deaths and 50 million injuries from road traffic accidents. It is the leading cause of death for the young. With the advent of mobile devices and mobile entertainment, this trend is only projected to increase. To reduce the distraction, the vehicle must first understand the distraction level of a user. In the past, many single-point on-body sensors and camera systems have been proposed to measure in car driver status (such as sleep, etc.) but these approaches are often limited to certain environments or require intrusive sensors on drivers that are difficult to deploy in reality.In this project, we will use inertial sensors embedded in the vehicle seat for recognizing driver’s distraction states. These includes physical distractions (such as texting and tuning the radio) and cognitivedistractions (including stress, fatigue, etc.). Advantages of the inertial sensor based driver monitoring system come from its simple and non-intrusive nature (i.e. no need for drivers to wear a device). We will build on the sensing platform developed with previous support from the University Transportation Center. We will focus our effort on developing data processing methods to extract detailed heart rate rhythm variability (for stress detection, etc.), and distraction related movement of drivers (for phoneusage detection, etc.). The main challenge resides in high noise due to moving vehicles and sensing location constraints. To address these challenges, we plan to utilize multiple sensor nodes, high-resolution and high frequency data with hybrid modeling approach to minimize uncertainties in signal processing and obtain reliable information through modeling of data as well as vehicle and humanresponses. We will experiment in real-vehiclesduring driving conditions to ensure real-worldapplicability of our system.
Description
1. Problem Statement:Every year about 1.3 million people die and 50 million people injure from road traffic crashes [1]. In particular, traffic related deaths is the leading cause of death among the young world-wide and is projected to continue to increase. Among many causes for road traffic injuries, driver distraction has been identified as an increasing concern for policy makers and researchers, including the usage of mobile phones and other technologies. Recently, the National Safety Council’s studies have shown that smartphones are responsible for 26% of these accidents [2, 3].However, comprehensive monitoring of driver distraction without further interference is a challenging task. Causes of driver distraction is a complicated process and is categorized into four different types: visual, auditory, cognitive, and physical. Combination of more than one type can happen simultaneously, triggered by either internal (in vehicle) or external sources of distraction [1]. Prior work has explored camera, or on-body sensors to monitor and maintain a particular type of distraction of the driver [4-8]. These works often have sensing requirements that require direct contact or line-of-sight with the driver, making them unsuitable for casual drivers and can in cases increase driver distraction. In addition, these system sense a pre-determined effects, while other effects are hidden.While the causes of distraction are various, the response of the driver tends to be physical (changing radio stations, leg placements, etc.) or cognitive (stressed, inattention, etc.). By capturing the physiological stateof the driver, we can detect both physical and cognitive distraction of the driver. Building on our prior work using inertial sensors [4, 8], we propose to measure physiological states using inertial sensors that are embedded into car seats. Specifically, we will develop data analysis methods to 1) extract detailed driver’s physical states (including movement, cardiovascular functions) and 2) infer cognitive states (including stress, fatigue, and attention level), under various driving scenarios. The main challenge resides in high noise level due to the moving vehicle and sensing constraints relying only on contacts. To address these challenges, we plan to utilize signal processing for multi-sourced, high-resolution and high frequency data with hybrid modeling approach to minimize uncertainties and obtain reliable information.2. Proposed Approach:The proposed research consists of three thrusts: 1) Separating the driver induced vibration from other car noise using multi-resolution signal analysis; 2) inferring driver’s physical states (including movement, cardiovascular functions) and 3) inferring cognitive states (including stress, fatigue, and attention level), under various driving scenarios. More details for each thrust is provided below.We developed an inertial-based sensing platform with previous support from the University Transportation Center (UTC). The platform consists of a network of small vibrational sensors distributed inside car seats to monitor the driver. Figure 1 shows the backside of a seat cushion with the sensing platform mounted. Advantages of the inertial sensor based driver monitoring system come from its simple and non-intrusive nature, which makes deployment possible. Based on the developed platform, we were able to successfully measure posture and breathing of drivers in noisy scenarios. Signal separation:The goal is to decompose vibration signals collected from inertial sensors into signal components each of which are induced by different sources, such as the driver’s gross movement, heartbeat, vehicle vibration, etc. The main challenge is that the sensed signal is a complex mixture of multiple signal components, in which the driver induced signal component to extract out is oftena weak signal with small amplitude overwhelmed by other car noise. Further, these mixture characteristics change over time depending on the driver’s state, which makes the separation difficult. To overcome the challenge, we propose to use a multi-resolution signal decomposition approach and causality analysis of multiple pairs of sensors to obtain the spatial distribution of the signal time-frequency spectrum. Based on the spectrum over spatio-temporal and frequency domains, we can identify signal components that are periodic (e.g., heartbeat), sporadic (e.g., posture change, speed bumps), and constant (e.g., ambient noise). Our prior work has shown that wavelet was particularly useful for detecting events of small amplitude from noisy signals [9].Physical State Inference: This thrust will extract driver’s distraction related movements and cardiovascular states so cognitive state of the driver can be inferred in the next module. Based on the physical model of human movement, we will investigate the causality between data from spatially distributed sensor nodes to estimate the movement. In our previous study, we demonstrated detecting the presence and posture of the driver whilst in the driver seat using time-frequency spectrum (Figure 2) and spatial distribution of signal energy (Figure 3), respectively. Building on this method, we will utilize causal analysis between each sensor pairs to extract spatial relationship between the driver induced vibrations in different locations (i.e., wave propagation pattern in the seat). The posture changes that are specifically related to the drivers’ distraction level will be explored and their states before and after each movement will be inferred from signal causality distribution and the physiology of human body.These movements incude 1) leaning forward/sideways, 2) extending arms for picking up or controlling other objects, and 3) periodic wobbling, etc.Heartbeat of a person is another good representation of the physical state of a driver. The heartbeatalso exerts vibration on the car seat that is small and periodic. Thus, we will identify a set of frequency domain features from the expected hear rate range from each axis of the accelerometer located near the driver’s heart in the seat back. The heart rhythm will be estimated by reconstructing the signal with detected frequency components. However, the noise of the car and body motion will often overwhelm the heartbeat signal. Therefore, the system must be able to model the motion noise as well as the heartbeat. A second observation from our prior modeling approach is that the sensed signal from other noise (e.g., car, body motion, etc.) will result in higher amplitude “outlier” sensor values, compared to the sensed signal during no significant noise. By modeling the distribution of the “outliers” and eventually selectively eliminating a majority of these higher/extreme segments that experience high motion noise, the system can reduce the effect of high noise level in the accelerometer signal. Building on our prior work on outlier analysis using extreme value distribution [10], we will model the high motion noise and perform an outlier detection to remove noise. The results will then be combined with multiple sensors to improve the stability of our system.Cognitive State inference: This thrust will extract driver’s cognitive distraction, through measures such as stress, fatigue, and attention level, based on the inferred physical states. For example, the variation of heart rate can be used to determine stress and thus cognitive distraction [11]. By determining heart beats and modeling the power spectrum of the heart-rate variability using machine learning, we can then infer stress. We will improve heartbeat detection accuracy to determine this variability. Similarly, motion can further suggest load that are due to different tasks or physical fatigue. These play a large role in determining attention ability. By utilizing sensors placed near the leg, we will infer the fatigue level of the driver especially when the muscles are tensed. Furthermore, the system can infer response time and position of the body for driving position analytics. We will take this work and utilize the model of the noise developed in the previous thrust to separate different distractive types of motion to extract attention level.3. Evaluation:We will evaluate our system in a realistic situation during driving scenarios. In particular, we will work on two main experimentation scenarios: simulated driving and in vehicle real-world driving. We will first test driving stress, noise model and noise separation algorithms efficiency in a controlled and safe lab environment. In addition, we will utilize the research Nissan Leaf vehicle at the lab to test our algorithm with the subjectin the passenger seat on isolated roads in the NASA research center where CMU SV campus is located. This will provide a more realistic noise environment to validate the algorithms we will develop. We will consider various driving scenarios, including different speed levels, stop and go, local road/highway/congestions, etc.
Timeline
Year 1 Develop noise removal and signal separation model for driver’s physical status (i.e. using mobile phone). Laboratory experiments will be conducted to evaluate the performance of the algorithm when drivers are well seated. Year 2 Develop and evaluate algorithms for in-vehicle movement and heartbeat variation recognition. Various models of human body will be incorporated into the system and evaluated to improve heart and motion state detection. Year 3 Extend algorithms for in-vehicle finer-grain driver physical and cognative state recognition, including fatigue, stress, and attention level. The system will include algorithms that separate the driving noise of the car and road, and tested for varying driving scenarios.
Strategic Description / RD&T
Deployment Plan
We are developing our algorithms based on the hardware outcomeof the previous year’s research, the system of a network of accelerometers distributed inside car seats. Initially, we will embed and test our algorithms in the car seat in our laboratory settings. This will serve as initial data collection and iterative algorithm development purposes. As the project progresses, we will test our algorithm in various driving scenarios through test-drive with a Nissan leaf for more realistic and on-road testing.
Expected Outcomes/Impacts
The outcome of our research will include a sensing system to infer driver’s distraction states in a car setting based on vibration sensors. In addition to the hardware developed from previous years, we will deliver embedded software algorithms to determine and classify the features of the person’s movements and cardiovascular activities, while removing noise and then infer the driver’s stress, attention and fatigue levels. This will help recognizing driver distraction levels in a car to make appropriate feedbacks to them in order to increase safety.We will evaluate the performance of the in-car driver monitoring algorithm in both simulated and real-world driving scenarios. The driving scenarios will include various situations such as low speed city driving, stop and go driving, idle, speed bump, high speed freeway driving, etc. The evaluation metrics will include movement and cardiovascular activity detection accuracy, false positive and false negative rates, robustness to noise and different driving scenarios.
Expected Outputs
TRID
Individuals Involved
| Name | Affiliation | Role | Position | |
|---|---|---|---|---|
| noh@cmu.edu | Noh, Hae Young | CEE | PI | Faculty - Untenured, Tenure Track |
| peizhang@cmu.edu | Zhang, Pei | ECE | Co-PI | Faculty - Research/Systems |
Budget
Amount of UTC Funds Awarded
$127500.00Total Project Budget (from all funding sources)
$127500.00Documents
Match Sources
No match sources!
Partners
No partners!