Principal Investigator

Maxine Eskenazi

Status

Completed

Start Date

Jan. 1, 2015

End Date

Dec. 31, 2015

Project Type

Research Advanced

Grant Program

MAP-21 TSET - Tier 1 (2012 - 2016)

Grant Cycle

2015 TSET UTC

Visibility

Public

Abstract

It is well known that drivers who talk on their phones become distracted, putting themselves and others  at  risk.  Despite  legislation,  people  still  use  their  phones  while  driving,  often  feeling  that they  are  in  control  of  the  situation.  Our  solution is to monitor the drivers’ speech and other information for evidence that they are distracted and then take appropriate action.

Using a driving simulator, the Distracted Driving Project has gathered speech and driving data  in  conditions  that  vary in their  degree  of  cognitive  load.The  speech  and  driving  data  are time-synced  so  that  coincidences  of  events  in  speech  and  high  cognitive  load  in  driving  can  be modeled.With funding from Yahoo!, we have mined this data to build a model and, from that, a set of  algorithms  to  create  a  first  distraction  detector.  This  detector  listens  to  the  driver conversing on the phone and, from information present in the speech signal alone, detects when the driver is becoming distracted. 

In  order  to increase  robustness and  avoid  false  alarms,  the  detector  needs to  model a larger  set  of  variables, which  include  information  about the  driving  environment.  For  this,  we will add information from the accelerometer, and other non-speech audio signals.From this, we will  create  anew  model  and corresponding set  of  algorithms.  This  new  robust  detector  is intended  to  make everyday  use  of  a  distraction  detector by  the  general  public  much  more acceptable. We  deliberately do  not  include any  of  the  car  system  sensors  in  our  detection algorithm since integration  of  phone  and  car  systems  will  often  not  be  available.  This  is especially the case for younger drivers with older model cars.

The second step in gaining drivers’ trust is to determine what the system should do when it decides that the driver is distracted. A human passenger conversing with a driver can be aware of the difficulty of the driving situation and modify their discussion with the driver appropriately. We will investigate  how  a  phone  app  can  be  driving-situation  aware and reactive in  a  similar way.  Potential strategies are simply stopping the app, or explicitly indicating that it is going to stop through  explicit  speech  or  implicit  signaling.We  will  identify  the  most  effective and graceful methods for a system to disengage from the dialog.    

Description

This past year, the Distracted Driving Project took the first steps in modeling distraction by creating a speech database that can be used to model normal and distracted driver speech. We set up a driving simulator connected to a Wizard of Oz setup that imitated a phone app reading email  to  a  driver.  The  driving  conditions  varied in  difficulty from  an  easy  straightaway  to  soft curves to tight curves. At the same time, the emails being read to the driver varied from easy to deal with (Mom asks how you are) to more challenging, requiring more of the driver’s cognitive load (John wants to know five things you would like to get for your birthday, he’ll choose one). Fifty  drivers  will  have  been  recorded  by  the  end  of  2014. A recorded sample  of  low  cognitive load on a straightaway can be found at https://dialrc.org/tim_control. A recorded sample of high cognitive  load  in  a  tight  curve  can  be  found  at https://dialrc.org/tim_hesit. The  data  we  have collected  so  far  clearly  shows detectable different  speech  behavior in  easier  and  more difficult driving situations.

The data, with funding from Yahoo!, is being modeled to represent control and distracted conditions uniquely from information in the speech signal time synced to the driving conditions. For  this  we  a reusing  automatic  speech  processing  tools  such  as  OpenSmile. Based  on  these models, we have begun to derive detection algorithms that are performing correctly on the data we have collected so far. 

While performance (precision in distraction detection based on speech) has been our first step, in order to have a detection system that drivers will actually accept to use takes two more steps. First  we  need to make  the  system  more  robust so  that  false  alarms  are  minimized.  The second  is  to determine  what action  is  appropriate  for  the  system to take when it detects distraction.This is the goal for our UTCFY2015 project.

In  order  to make  the  system  more  robust,  we will complement  the speech  signal information  with  non-speech  data.  Accelerometer information  can  be  obtained from any smartphone. Sudden  accelerations  decelerations may  be  indicators  of  a  dangerous  driving condition.  These  may  take  place while  a  driver  is  speaking,  at  a  time  they  are  not  speaking, and/or  before  distraction shows  up  in  the  driver’s speech. Other types  of  information  such  as GPScan also help increase the detector’s robustness.GPS information can be used to tell us the car’s speed and location relative to some areas where driving is difficult. We  will record  and several  types  of  signals such  as  these  and  include  them in  our  models  and  algorithms,  finally testing them  for  relative  effectiveness. This  will  entail collecting  a  new  dataset  from  both the driving simulator we have already set up and aboard real vehicles(in controlled conditions that do not  present  a  danger  to  the driver).  We  will  test  whether  the  addition  of  each  of the  signals together and individually contributes to a decreased false positive rate and modify our algorithms accordingly.This will make our system more robust to false alarms, a strong first step in gaining drivers’ trust and thus adoption of the system. 

The second step concerns system interaction with the driver. Here we will discover how to gracefully intervene when distraction is detected. Not all types of intervention that we create will  be  acceptable  to  the  average  driver. In  our  Yahoo!  project,  we are  creating a  system  that reads the driver’s email to them(they can dictate answers to the mail as well). The goal here is to have the  detection system gracefully shut  down  the  mail app.  That  is,  the  system  stops  reading mail  or  listening  to  the  driver  dictate  mail  and  saves  the  last  few  actions  that  happened  in  the dialog  so  that  the  driver  can  come  back  to  exactly  what  they  were  doing  with  no  loss  of information  at  a  later  time. Yet  there  is  no  proof  that having  the  system  rather  than  the  driver decide when to shut down the application is acceptable to drivers in general This may in fact be more distracting if,  for  example, the  driver ends  up paying  more  attention  to wondering  what happened to the app.

There  are  several possible actions that we can  set up  in  a  smartphone-based  system/app that can be taken when distraction is detected: 
- the app can simply be shut down gracefully; 
- the system can activate some warning alarm; 
- the  system  can send  one  or more successive  messages  to  the  driver  either  aurally visually or in some tactile form (vibrating steering wheel, for example). The first solution involves a system-driven decision about what to do and when. The two other solutions give the driver the choice. 

We  will  test  the  acceptability  of several  versions  of each  of these options.  While acceptability  is  the  main  concern,  we  will  also  measure  whether  each option  tested adds to  the driver’s cognitive load, thus distracting them further. We will also investigate whether we should offer only one solution for all drivers or if the driver should be given the choice of the type of warning they prefer. Finally, we will test these options for acceptability and added cognitive load both with our driving simulator and in a real vehicle.

We will employ one LTI MLT graduate student, Tiancheng Zhao, for this project.

Timeline

One year.

Strategic Description / RD&T

Deployment Plan

We have recently met with representatives of Mercedes. While they were very interested in our work, they cannot work with us yet. To partner with us, they need proof that the detection works robustly and that it is acceptable to drivers. The results of this UTCFY2015 project will enable us to approachcar manufacturers and others to find a means to deploy our system.The  distraction  detection  that we  havedevelopedwith  Yahoo!  specifically  for  their  newsfeed  and mailapps should be deployed some time in 2015.Our techniques are specifically designed to use whatever information is available, thus if we only have an audio stream (through the telephone system) we can use that stream to detect distraction and reactappropriately.  If we have an app on the driver’s smartphone we can use sensors on the phone itself to aid indetection.  Finally,if our system is integratedwith the car's sensors we can also utilize them.

Expected Outcomes/Impacts

- Two sets of driving data enriched with accelerometer and other information.
- A  robust  distraction  detection  algorithm. After  a  series  of  offline  tests,  it will  be  tested  on  the  data  we have  collected  by comparing  it  to human  annotation  of the  degree  of  the  driver’s distraction  and determining whether the system performs similarly to the humans. We will measure the Cohen Kappa of agreement using tenfold cross-validation.The system will also be tested for correct performance in both a simulator and a real vehicle. The driver will be asked to assess system performance.
- A  system  that gracefully  alerts  drivers that  they  are  distracted.  This  will  be  assessed  by  having  drivers use the system with the simulator and in real vehicles to see if the types of alarms are acceptable to them –if they would use them. Fifty drivers will be enlisted for this task.

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
awb@cs.cmu.edu	Black, Alan	LTI	Co-PI	Faculty - Research/Systems
max@cs.cmu.edu	Eskenazi, Maxine	LTI/SCS	PI	Faculty - Research/Systems
tingyaoh@andrew.cmu.edu	Hu, Ting Yao	MLT	Other	Student - Masters
tianchez@andrew.cmu.edu	Zhao, Tiancheng	MLT	Other	Student - Masters

Budget

Amount of UTC Funds Awarded

$100000.00

Total Project Budget (from all funding sources)

$100000.00

Documents

Type	Name	Uploaded
Final Report	UTC_project_83_Detecting_Distraction_Final_report.pdf	July 2, 2018, 4:33 a.m.

Match Sources

No match sources!

Partners

No partners!