#47 Non-Intrusive Driver Distraction Monitoring Using Vehicle Vibration Sensing

Principal Investigator
Hae Young Noh
Start Date
Jan. 1, 2017
End Date
Aug. 31, 2018
Research Type
Grant Type
Grant Program
MAP-21 TSET National (2013 - 2018)
Grant Cycle


Driver  distraction  is  responsible  for  more  than  a  quarter  of  the  1.3  million  deaths  and  50  million  injuries from  road  traffic  accidents.  It  is  the  leading  cause  of  death  for  the  young.  With  the  advent  of  mobile devices and mobile  entertainment, this trend is only projected to increase. To reduce the distraction, the vehicle  must  first  understand  the  distraction  level  of  a  user.  In  the  past,  many  single-point  on-body sensors  and  camera  systems  have  been  proposed  to  measure  in  car  driver  status  (such  as  sleep,  etc.) but these approaches are often limited to certain environments or require intrusive sensors on drivers that are difficult to deploy in reality.In  this  project,  we  will  use  inertial  sensors  embedded  in  the  vehicle  seat  for  recognizing  driver’s distraction  states.  These  includes  physical  distractions  (such  as  texting  and  tuning  the  radio)  and cognitivedistractions  (including  stress,  fatigue,  etc.).  Advantages  of  the  inertial  sensor  based  driver monitoring  system  come  from  its  simple  and  non-intrusive  nature  (i.e.  no  need  for  drivers  to  wear  a device).  We  will  build  on  the  sensing  platform  developed  with  previous  support  from  the  University Transportation Center. We will focus our effort on developing data processing methods to extract detailed heart  rate  rhythm  variability  (for  stress  detection,  etc.),  and  distraction  related  movement  of  drivers  (for phoneusage  detection,  etc.).  The  main  challenge  resides  in  high  noise  due  to  moving  vehicles  and sensing location constraints. To address these challenges, we plan to utilize multiple sensor nodes, high-resolution  and  high  frequency  data  with  hybrid  modeling  approach  to  minimize  uncertainties  in  signal processing  and  obtain  reliable  information  through  modeling  of  data  as  well  as  vehicle  and  humanresponses. We will experiment in real-vehiclesduring driving conditions to ensure real-worldapplicability of our system.    
1. Problem Statement:Every year about 1.3 million people die and 50 million people injure from road  traffic  crashes  [1].  In  particular,  traffic  related  deaths  is  the  leading  cause  of  death  among the  young  world-wide  and  is  projected  to  continue  to  increase.  Among  many  causes  for  road traffic  injuries,  driver  distraction  has  been  identified  as  an  increasing  concern  for  policy  makers and  researchers,  including  the  usage  of  mobile  phones  and  other  technologies.  Recently,  the National Safety Council’s studies have shown that smartphones are responsible for 26% of these accidents [2, 3].However,   comprehensive   monitoring   of   driver   distraction   without   further   interference   is   a challenging  task.  Causes  of  driver  distraction  is  a  complicated  process  and  is  categorized  into four different types: visual, auditory, cognitive, and  physical. Combination  of more than one type can  happen  simultaneously,  triggered  by  either  internal  (in  vehicle)  or  external  sources  of distraction  [1].  Prior  work  has  explored  camera,  or  on-body  sensors  to  monitor  and  maintain  a particular type of distraction of the driver [4-8]. These works often have sensing requirements that require  direct  contact  or  line-of-sight  with  the  driver,  making  them  unsuitable  for  casual  drivers and  can  in  cases  increase  driver  distraction.  In  addition,  these  system  sense  a  pre-determined effects, while other effects are hidden.While  the  causes  of  distraction  are  various,  the  response  of  the  driver  tends  to  be  physical (changing  radio  stations,  leg  placements,  etc.)  or  cognitive  (stressed,  inattention,  etc.).  By capturing  the physiological  stateof  the  driver,  we  can  detect  both physical and cognitive distraction  of  the  driver.  Building  on  our  prior  work  using  inertial  sensors  [4,  8],  we  propose  to measure   physiological   states   using   inertial   sensors that   are   embedded   into   car   seats. Specifically, we will develop data analysis methods to 1) extract detailed driver’s physical states (including  movement,  cardiovascular  functions)  and  2)  infer  cognitive  states  (including  stress, fatigue, and attention level), under various driving scenarios. The main challenge resides in high noise  level  due  to  the  moving  vehicle  and  sensing  constraints  relying  only  on  contacts.  To address  these  challenges,  we  plan  to  utilize  signal  processing  for  multi-sourced,  high-resolution and  high  frequency  data  with  hybrid  modeling  approach  to  minimize  uncertainties  and  obtain reliable information.2.  Proposed  Approach:The  proposed  research  consists  of  three  thrusts:  1)  Separating  the driver  induced  vibration  from  other  car  noise using  multi-resolution  signal  analysis;  2)  inferring driver’s physical states (including movement, cardiovascular functions) and 3) inferring cognitive states (including stress, fatigue, and attention level), under various driving scenarios. More details for each thrust is provided below.We  developed  an  inertial-based  sensing  platform  with  previous  support  from  the  University Transportation  Center  (UTC).  The  platform  consists  of  a  network  of  small  vibrational  sensors distributed inside car seats to monitor the driver. Figure  1 shows the backside of a seat cushion with  the  sensing  platform  mounted.  Advantages  of  the  inertial  sensor  based  driver  monitoring system come from its simple and non-intrusive nature, which makes deployment possible. Based on  the  developed  platform,  we  were  able  to  successfully  measure  posture  and  breathing  of drivers in noisy scenarios.
Signal  separation:The  goal  is  to  decompose  vibration  signals  collected  from  inertial  sensors into signal components each of which are induced by different sources, such as the driver’s gross movement,  heartbeat,  vehicle  vibration,  etc.  The  main  challenge  is  that  the  sensed  signal  is  a complex mixture of multiple signal components, in which the driver induced signal component to extract out is oftena weak signal with small amplitude overwhelmed by other car noise. Further, these mixture characteristics change over time depending on the driver’s state, which makes the separation  difficult.  To  overcome  the  challenge,  we  propose  to  use  a  multi-resolution  signal decomposition  approach  and  causality  analysis  of  multiple  pairs  of  sensors  to  obtain  the  spatial distribution  of  the  signal  time-frequency  spectrum.  Based  on  the  spectrum  over  spatio-temporal and  frequency  domains,  we  can  identify  signal  components  that  are  periodic  (e.g.,  heartbeat), sporadic (e.g., posture change, speed bumps), and constant (e.g., ambient noise). Our prior work has shown that wavelet was particularly useful for detecting events of small amplitude from noisy signals [9].Physical  State  Inference: This  thrust  will  extract  driver’s  distraction  related  movements  and cardiovascular states so cognitive state of the driver can be inferred in the next module. Based on the  physical  model  of  human  movement,  we  will  investigate  the causality  between  data  from spatially   distributed  sensor  nodes  to  estimate  the   movement.  In  our  previous  study,   we demonstrated  detecting  the  presence  and  posture  of  the  driver  whilst  in  the  driver  seat  using time-frequency   spectrum   (Figure   2)   and   spatial distribution   of   signal   energy   (Figure   3), respectively. Building on this method, we will utilize causal analysis between each sensor pairs to extract spatial relationship between the driver induced vibrations in different locations (i.e., wave propagation pattern in the seat). The posture changes that are specifically related to the drivers’ distraction level will be explored and their states before and after each movement will be inferred from signal causality distribution and the physiology of human body.These movements incude 1) leaning  forward/sideways,  2)  extending  arms  for  picking  up  or  controlling  other  objects,  and  3) periodic wobbling, etc.Heartbeat  of  a  person  is  another  good  representation  of  the  physical  state  of  a  driver.  The heartbeatalso exerts vibration on the car seat that is small and periodic. Thus, we will identify a set  of  frequency  domain  features  from  the  expected  hear  rate  range  from  each  axis  of  the accelerometer located near the driver’s heart in the seat back. The heart rhythm will be estimated by reconstructing the signal with  detected frequency components. However, the noise of the car and body motion will often overwhelm the heartbeat signal. Therefore, the system must be able to model  the  motion  noise  as  well  as  the  heartbeat.  A  second  observation  from  our  prior  modeling approach  is  that  the  sensed  signal  from  other  noise  (e.g.,  car,  body  motion,  etc.)  will  result  in higher amplitude “outlier” sensor values, compared to the sensed signal during no significant noise.  By modeling  the  distribution  of  the  “outliers”  and  eventually  selectively  eliminating  a majority  of  these  higher/extreme  segments  that  experience  high  motion  noise,  the  system  can reduce  the  effect  of  high  noise  level  in  the  accelerometer  signal.  Building  on our  prior  work  on outlier  analysis  using  extreme  value  distribution  [10],  we  will  model  the  high  motion  noise  and perform  an  outlier  detection  to  remove  noise.  The  results  will  then  be  combined  with  multiple sensors to improve the stability of our system.Cognitive   State   inference: This  thrust  will  extract  driver’s  cognitive  distraction,  through measures such  as stress, fatigue, and  attention level, based  on  the inferred physical states. For example, the variation of heart rate can be used to determine stress and thus cognitive distraction [11].  By  determining  heart  beats  and  modeling  the  power  spectrum  of  the  heart-rate  variability using machine learning, we can then infer stress. We will improve heartbeat detection accuracy to determine this variability.
Similarly,  motion  can  further  suggest  load  that  are  due  to  different  tasks  or  physical  fatigue. These  play  a  large  role  in  determining  attention  ability.  By  utilizing  sensors  placed  near  the  leg, we will infer the fatigue level of the driver especially when the muscles are tensed. Furthermore, the system can infer response time and position of the body for driving position analytics. We will take  this  work  and  utilize  the  model  of  the  noise  developed  in  the  previous  thrust  to  separate different distractive types of motion to extract attention level.3.  Evaluation:We  will  evaluate  our  system  in  a  realistic  situation  during  driving  scenarios.  In particular,  we  will  work  on  two  main  experimentation  scenarios: simulated  driving  and  in  vehicle real-world  driving.  We  will  first  test  driving  stress,  noise  model  and  noise  separation  algorithms efficiency in a controlled and safe lab environment. In addition, we will utilize the research Nissan Leaf  vehicle  at  the  lab  to  test  our  algorithm  with  the  subjectin  the  passenger  seat  on  isolated roads in the NASA research center  where CMU SV campus is located. This  will provide a more realistic  noise  environment  to  validate  the  algorithms  we  will  develop.  We  will  consider  various driving  scenarios,  including  different  speed  levels,  stop  and  go,  local  road/highway/congestions, etc.    
Year 1 Develop noise removal and signal separation model for driver’s physical status (i.e. using mobile phone).  Laboratory  experiments  will  be  conducted  to  evaluate  the  performance  of  the  algorithm when drivers are well seated.

Year 2 Develop  and  evaluate  algorithms  for  in-vehicle  movement  and  heartbeat  variation  recognition. Various  models  of  human  body  will  be  incorporated  into  the  system  and evaluated  to  improve heart and motion state detection.

Year 3 Extend  algorithms  for  in-vehicle  finer-grain  driver  physical  and  cognative  state  recognition, including fatigue, stress, and attention level. The system will include algorithms that separate the driving noise of the car and road, and tested for varying driving scenarios.    
Deployment Plan
We  are  developing  our  algorithms  based  on  the  hardware  outcomeof  the  previous  year’s research, the system of a network of accelerometers distributed inside car seats. Initially, we will embed and test our algorithms in the car seat in our laboratory settings. This will serve as initial data collection and iterative algorithm  development purposes.  As the  project progresses,  we  will test  our  algorithm  in  various  driving  scenarios  through  test-drive  with  a  Nissan  leaf  for  more realistic and on-road testing.    
Expected Accomplishments and Metrics
The outcome of our research will include a sensing system to infer driver’s distraction states in a car  setting  based  on vibration  sensors.  In  addition  to  the  hardware  developed  from  previous years, we will deliver embedded software algorithms to determine and classify the features of the person’s  movements  and  cardiovascular  activities,  while  removing  noise  and  then  infer  the driver’s stress, attention and fatigue levels. This will help recognizing driver distraction levels in a car to make appropriate feedbacks to them in order to increase safety.We will evaluate the performance of the in-car driver monitoring algorithm in both simulated and real-world driving scenarios. The driving scenarios will include various situations such as low speed city driving, stop and go driving, idle, speed bump, high speed freeway driving, etc. The evaluation metrics will include movement and cardiovascular activity detection accuracy, false positive and false negative rates, robustness to noise and different driving scenarios.    

Individuals Involved

Email Name Affiliation Role Position
noh@cmu.edu Noh, Hae Young CEE PI Faculty - Untenured, Tenure Track
peizhang@cmu.edu Zhang, Pei ECE Co-PI Faculty - Researcher/Post-Doc


Amount of UTC Funds Awarded
Total Project Budget (from all funding sources)


Type Name Uploaded
Publication Heart and Sole: Shoe-based heart rate monitoring Sept. 30, 2017, 12:02 p.m.
Presentation Structures as Sensors: Indirect Monitoring of Humans and Surrounding Sept. 30, 2017, 12:02 p.m.
Presentation Structures as Sensors: Indirect Monitoring of Humans and Surrounding Sept. 30, 2017, 12:02 p.m.
Presentation Structures as Sensors: Indirect Monitoring of Humans and Surrounding Sept. 30, 2017, 12:02 p.m.
Progress Report 47_Progress_Report_2016-10-01 Sept. 30, 2017, 12:02 p.m.
Progress Report 47_Progress_Report_2017-09-30 Oct. 5, 2017, 10:02 a.m.
Publication VVRRM: Vehicular Vibration-based Heart RR-Interval Monitoring System March 31, 2018, 7:10 p.m.
Presentation Structures as Sensors: Indirect Monitoring of Humans and Surrounding March 31, 2018, 7:10 p.m.
Presentation VVRRM: Vehicular Vibration-based Heart RR-Interval Monitoring System March 31, 2018, 7:10 p.m.
Progress Report 47_Progress_Report_2018-03-31 March 31, 2018, 7:10 p.m.
Final Report TSET_Final_Report_-_47_Non-Intrusive_Driver_Distraction_Monitoring_Using_Vehicle_Vibration_Sensing.pdf Oct. 8, 2018, 4:36 a.m.
Publication An information-theoretic approach for indirect train traffic monitoring using building vibration. Dec. 2, 2020, 9:42 a.m.
Publication Vehicular vibration-based heart RR-interval monitoring system. Dec. 2, 2020, 9:44 a.m.
Publication Seat vibration for heart monitoring in a moving automobile. Dec. 8, 2020, 9:52 a.m.

Match Sources

No match sources!


No partners!