Principal Investigator

Hae Young Noh

Status

Completed

Start Date

Jan. 1, 2015

End Date

Dec. 31, 2015

Project Type

Research Advanced

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

2015 TSET UTC - National

Visibility

Public

Abstract

Autonomous vehicles are rapidly defining the automotive industry today. However, the transition from today's  manned  autos  to  the  future's  autonomous  vehicles  poses many  challenges.  Unexpected situations (such as snow, unmarked roads, etc.) or manual requests will cause the car to give control back to the human driver. Before the car can hand control back to the driver, the car needs to know the state of the driver, such as attention level, fatigue, and stress. Many single-point on-body sensors and camera systems have been proposed, but these approaches are often limited to controlled laboratory environments or require intrusive sensors on the driver that are difficult to deploy in reality. Additional challenges arise in terms of the impact of vehicle dynamics on sensing noise and constraints on sensor placements.

In this project, we will investigate vehicle-based inertial sensors for recognizing driver’s physiological states including posture, movement, muscular activity, and cardiovascular functions. Advantages of the inertial sensor based driver monitoring system come from its simple and non-intrusive nature, which makes deployment easy. A network of small vibrational sensors distributed inside car seats and seat belts will measure vibration induced by the driver’s body. We will then use these vibrations to infer higher level human states, such as stress level, restlessness, and fatigue. The main challenge resides in the high noise level due to the moving vehicle and sensing constraints relying only on contacts. To address these challenges, we plan to utilize multi-sourced, high resolution and high frequency data with multi-resolution signal processing techniques to reject noise and obtain higher level directed information. We will also incorporate physical models of human body and vehicle dynamics in addition to data-driven models. For validation, we are working with our industry partner (Renault) to build our system and embed into cars for laboratory and field testing.    

Description

A.  Introduction:  Autonomous  vehicles  are  becoming  a  reality.  Many  car  companies  are  already incorporating advanced cruise control systems (such as active lane-keep, automatic breaking, etc.) into current  generation  of  vehicular  systems.  As  cars  become  more  autonomous,  incidences that  require human driver involvement will become less [1-3]. However, this will likely result in less attentiveness in the drivers when driver involvement is required. In order for the car to safely give control to the driver, the system must be able to understand the attention level of the driver. 

Prior work has explored on-body sensors to maintain attention level of the driver [4-7]. These works often have sensing requirements that require direct contact with the driver, making them unsuitable for casual drivers. Another approach utilizes camera based systems that monitor the driver [5, 7]. These systems are often sensitive to different lighting and line-of-sight limitations. Furthermore, these works focuses on maintaining the driver's attention, as oppose to understand the level of inattention due to the current driver state. 

In order to address the challenge of understanding attention level, we propose a non-intrusive sensing system embedded in the car seats to infer driver physiological states in two levels: 1) posture and motion (macro-motion) and 2) muscular and cardiovascular states (micro-motion). Based on this information, we will obtain higher level understanding of the driver’s state, such as attention level, stress, and fatigue. In particular our system will have the following key features:
•    Inference-based sensing hardware system to enable non-intrusive in-car deployments.
•    Non-intrusive measurement of driver motion and physiological information.
•    Combining signal processing and machine learning (data-driven) with models of automotive vibrations and human body (physics-based) to further distinguish signals of interest.

B.  System  Overview: Our  driver  monitoring  system  will  be  built  on  modular  hardware we  have developed as part of prior work [4, 8]. We foresee a system that consists of three-tiers: 1) an inertial sensor node network to sense motion and muscle vibration, 2) a mobile data aggregator to collect and transmit  data,  and  3)  a  backend  server  to  process  transmitted  sensor  data.  Each  tier  has  different processing and communication capabilities that must be dynamically optimized over different sensing applications. A brief overview of these three main components is shown in Figure 1, and their details are provided below.

Sensor Node Network: The sensor network of the system is made up of a set of small unobtrusive inertial sensor nodes that enable fine-grained activity monitoring by detecting body motion as well as the skeletal muscle vibrations. The network consists of inertial sensor nodes, a micro controller chip, and three triaxial sensors: accelerometers, gyroscopes, and magnetometers. The main challenge in sensing is that since the sensor nodes are designed to be small and to collect data as fast as possible to capture small amplitude transient vibrations, they are relatively resource constrained. To this end, we will use a mobile data aggregator, to coordinate their sampling and data transmission to the backend server. However, the communication bandwidth is limited and the number of sensors per seat are high (20+). Thus, we will explore the tradeoff between different sampling rate, processing time, and communication bandwidth between the sensors and the aggregator. We will also explore the processing and operation of low-level compression  and  feature  extraction  on  the  nodes  and  their  impacts  on  the  backend  server  data processing. 

Mobile Data Aggregator: Since the sensor nodes are resource constrained, the monitoring system uses a mobile data aggregator to coordinate the sensor sampling, error correction, and wireless data transfer to the backend server. In other words, this component is in charge of driving the sensor network, as the master on the node system. We will build on our prior work on the hardware developed on a bodysuit that used  a  wired  connection  to  communicate  with  the  other  sensors  on  the  same  suit  and  Wi-Fi  to communicate with the base station.  Although the network is extremely bandwidth limited due to the high number of sensors, the processing on the aggregator is less constrained compared to the sensors. Thus we  will  explore  asymmetrical  compression  and  processing  techniques  both  between  sensors  and aggregator as well as aggregator and back-end server. 

Backend  Server:  The  backend  server  receives  inertial  data  to  create  human  motion  and  vibration signatures to infer muscle identity as well as stress level. The back end server contains two modules: 1) posture and motion recognition (macro-motion), and 2) muscular and cardiovascular activity inferencing (micro-motion) module. The first module involves detecting large amplitude low frequency motions, while the latter one focuses on small amplitude high frequency signals. We aim to facilitate a minimal system setup and motion capture without tagging sensor location and orientation. Initially we will use statistical signal processing and machine learning algorithms to recognize specific motions of interest and identify activated  muscles.  Then  we  will  combine  them  with  analytical  models  of  human  body  and  vehicle dynamics to further improve the inference and obtain physical understanding of the driver’s physiological status. 

The posture and motion recognition module will extract overall activities of the driver through human body modeling and inertial data mining. The main challenge of this module is to deduce those information using limited sensing ranges (i.e., only through the contacts). To overcome this limitation, we will combine data from multiple sources and utilize a physical model of human body to compensate for unobserved body posture and motion. In our prior work, we deployed a 20 accelerometers system on a cushion and attach the cushion on a test seat, on which a driver applied different movements sitting. Figure 2 shows the results when the driver shifts breathing rate. We observed a high correlation between the time-frequency domain features extracted from the sensor data and driver's movement shifts, which shows a promising potential for recognizing driver states using vibration analysis.

The muscular and cardiovascular activity inferencing module identifies activated muscles and measures muscle tension and cardiac muscle activities (heart rate, blood flow, etc.), which in turn are used to infer stress and fatigue level of the driver. This fine-grained activity recognition will be achieved by combining feature selection and machine-learning techniques to achieve muscle group identification. We will then use the change in the data to determine states of the muscle, namely to differentiate between a fresh and fatigued muscle. The human heartbeat generates a rather sizable vibration, which can also be detected using our sensors. While current work has looked at using heart-rate for monitoring fitness, we will be looking into detecting the beat from multiple sensors and inferring abnormalities in heart beats such as murmurs, and irregular heartbeats, using our system. This approach will allow us to provide an expandable system, adaptable to other in-vehicle applications such as heart monitoring and failure prevention. The key challenges in obtaining finer granularity body data, such as inferring muscle fatigue and blood flow, resides in low signal to noise ratio, where the signal of interest is very low amplitude, while others such as motion and vehicle dynamics result in large amplitude signals. In fact, the muscular and cardiovascular vibrations have overlapping ranges as motion and vehicle dynamics. We plan to initially infer this information only when detected motion is low combining motion tracking and activity information to determine when the lower frequency component has the most likelihood of success. Then, we will model vehicle dynamics to separate out the signal of interest. We will explore the feasibility of measuring vibrations emerging from body fat content as stimulated by the dynamic motion of a driving vehicle. We will explore both the relative surface fat in certain areas as well as estimate total body shape base on the resonance of the outer body tissues.

C. Evaluation: One key aspect of the work is to validate its applicability in a driving scenario. We will design the system to work under a variety of non-driving and driving scenarios. We plan on testing under three setups. 1) Initially, we will gather data and test utilizing a separate car seat, and then test under controlled noise conditions. This will allow our initial data collection and algorithm development. 2) Second phase of our tests will be performed in a parked running car. This will provide initial models of the vehicle models and give controlled baselines to known motion. 3) Finally we will test the system in a Nissan Leaf test car with our industry partner (Renault). In particular we will test in the following conditions: high speed freeway driving, and low speed stop-go driving. For baseline data, the system will be tested with human subjects sitting in the car and performing a number of stress tests. These tests will include tensing muscles and performing tasks before driving. For physiological baseline, we will collect data using ECG and EMG equipment available in the lab. The particular of the tests will be determined pending IRB approval. 

Timeline

Jan. 1 - Mar. 30, 2015
Develop  hardware  that  consists  of  the  sensor  node,  aggregator, and  backend  server  to  collect  the physiological information in a car setting. This system will include the hardware, design documents and preliminary software algorithms to determine and classify the features of a person’s physiological state (in-vehicle muscle recognition).
Mar. 1 - May. 31, 2015
Develop and evaluate algorithms for in-vehicle motion and posture recognition. Various models of human body will be incorporated into the system and evaluated.
Jun. 1 - Oct. 31, 2015
Develop  and  evaluate  algorithms  for  in-vehicle  finer-grain  physiological  state  recognition,  including muscle activation, fatigue, heart rate, blood flow, and body fatness signal. The system will also include algorithms that separate the driving noise of the car and road.
Nov. 1 - Dec. 31, 2015
Deployment and validation in various driving scenarios. 

Strategic Description / RD&T

Deployment Plan

We are developing our system as a cushion that can be easily incorporated on top of the car seats for preliminary testing. This will serve as initial data collection and algorithm development purposes. As the project progresses, Renault has provided our project with a car seat for the Nissan Leaf for more realistic implementation. Furthermore, Renault research (co-located with Nissan research silicon valley and is within a 5 minute drive of the CMU silicon valley campus) will provide access to a test Nissan leaf for use at their research facility for more realistic and on-road testing as the research progresses. 

Expected Outcomes/Impacts

The outcome of our research will include a hardware that consists of the sensor node, aggregator, and backend server to infer driver’s physiological states in a car setting. This system will include both the hardware and software algorithms to determine and classify the features of the person’s macro-motions (posture  and  motion)  and  micro-motions  (muscular  and  cardiovascular  activities),  which  helps  the autonomous car with understanding the driver’s states, such as attention level, fatigue, and stress. The system will also include algorithms that separate the driving noise of the car and road.

We will evaluate In-vehicle data collection, testing, and demo for the in-car driver monitoring system under real-world driving scenarios. We will keep demo and test as well as collect data based on a number of driving scenarios. In particular we will test the following situations: High speed freeway driving, Low speed  stop-go  driving,  and  Idle.  The  evaluation  metrics  will  include  motion  and  activity  detection accuracy, false positive and false negative rates. 

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
noh@cmu.edu	Noh, Hae Young	CEE	PI	Faculty - Research/Systems
peizhang@cmu.edu	Zhang, Pei	Carnegie Mellon University	Co-PI	Faculty - Research/Systems

Budget

Amount of UTC Funds Awarded

$100000.00

Total Project Budget (from all funding sources)

$100000.00

Documents

Type	Name	Uploaded
Final Report	101_-_TSET_Final_Report_-_Driver_Status_Monitoring_in_Autonomous_Vehicles_Using_In-Seat_Inertia_Sensors.pdf	July 26, 2018, 6:49 a.m.

Match Sources

No match sources!

Partners

No partners!