Project: #286 Personalized Trip Planner for Seniors - GetGoing Progress Report - Reporting Period Ending: March 30, 2020 Principal Investigator: Maxine Eskenazi Status: Active Start Date: July 1, 2019 End Date: June 30, 2020 Research Type: Advanced Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: 2019 Mobility21 UTC Progress Report (Last Updated: June 30, 2020, 6:25 a.m.) % Project Completed to Date: 45 % Grant Award Expended: 45 % Match Expended & Document: 45 USDOT Requirements Accomplishments System State GetGoing has been developed into a fully functioning system. It has gone through several iterations of user feedback (from various sources including AARP, and most recently UPMC) and development. Much of the focus has been in ensuring that the system is accessible to older users. Some of the features and design decisions that make this possible are enumerated below: Confirmation of understanding. GetGoing confirms that the user understood the system output, and had sufficient time to write it down if they desire. This addresses the concern that older users be able to comprehend and retain instructions. This feature was strongly requested by AARP. Slower Synthesized Speech. By inserting pauses prior to synthesizing the speech signal, GetGoing helps to make its utterances more understandable for individuals who process speech more slowly. Attention-Grabbing Prefix. GetGoing inserts an attention-grabbing prefix prior to providing important information. This allows users to switch attention and tune out distractions and focus on the system output. Barge-In. Allowing users to interrupt the system shifts control of the dialog toward the user. Since senior users may be unfamiliar with voice-based assistants, barge-in allows them to correct system mistakes without having to wait for the system turn to finish. This avoids the user's turns getting "out-of-sync" with the system. Flexible Dialog Manager. GetGoing's dialog manager, based on the RavenClaw framework, is flexible in the order of turns. This flexibility is in part due to our generalized natural language understanding which is a deep learning model that relies on language embeddings pre-trained on large amounts of text (including all of Wikipedia). The flexible dialog manager allows users to easily correct the system, provide information out of turn, or fill multiple slots at once. This makes the dialog more natural. Telephone Connection. GetGoing's user interface is the telephone, which reduces the entry barrier for senior users who may not own a smartphone or may be uncomfortable using one. In its current state, users can call a publicly available phone number and ask the system for directions in Southwestern Pennsylvania. During the next reporting period we will be making the system more robust to automatic recognition errors. This in turn will give potential users more confidence in the system. Once the system has been tested and is proving to be more robust, we will continue our push with the following partners: AARP. UPMC, Senior News. This will help us get both focus groups and users. During the period ending 3-30-2020, we have: - installed a more complete list of bus stops, and more monuments in the Pittsburgh area - made the user experience better by: skipping explicit confirmation for recognition results with higher confidence refining instructions for first time users, with example usages and less repetition giving the system more robust audio speech recognition for time and location improving step chunking, so the repeat command now only repeats the latest step instead of including all the earlier instructions creating a more natural direction delivery, with no more long address and abbreviations in steps - improved the system by: adjusting the prosody output, with key information being more emphasized with a louder voice creating a separate development branch, and also a separate phone number for the all-user mode (no specific accommodations for seniors) fixing various bugs in time, prosody, dialog management retraining the Natural Language Understanding (NLU) model so now it recognizes more different ways of saying “continue”, “repeat”, and better answers yes/no questions - demoed GetGoing in November 2019 at the Mobility 21 showcase - began work on accessibility information giving: adding alternative routes to the UPMC Presbyterian campus, based on expert suggestions automating the translation of verbal directions into machine-usable directions and backend lookups (in progress) Providing a plan based on arrival time instead of departure time and oriented toward the final location (especially in hospitals) Impacts Tech Transfer Since GetGoing has been developed as a senior-friendly system, many of the features can be separated out into a layer. This layer could potentially be applied onto other existing spoken dialog systems to make them more accessible for the senior population. A disclosure of intellectual property for ABLE (AccessiBility LayEr) was submitted to the CMU Center for Technology Transfer and Enterprise Creation on June 13th, 2019. We had a meeting with representatives from the Center for Tech Transfer on August 19th, 2019. After this discussion, a provisional patent was filed on September 3rd, 2019. Paper A paper on the system, titled “CMU GetGoing: An Understandable and Memorable Dialog System for Seniors” (https://arxiv.org/abs/1909.01322) was recently presented at the Dialog for Good Workshop (DiGo). DiGo was held in Stockholm, Sweden on September 10th, 2019. It is a workshop at the SIGDial conference (Special Interest Group in Dialog). This paper describes the system, with specific attention to the features that improve accessibility for seniors. For this paper, a thorough user study was carried out, which demonstrates that the system is quantifiably better at ensuring that older adults understand and retain information. Other Get Going collected paid user data for the DiGo paper and has made it publicly available at https://github.com/DialRC/GetGoingData/. Outcomes New Partners UPMC PathVu Issues The goal of the coming period of work has been to collect as much data from real users as possible and then to use to personalize the system. Since the advent of the Covid19 pandemic, people, especially seniors, are not traveling. Instead of data collection and personalization, we are working on precise instructions to hospitals (as described above) and giving information about the best path to take to a given destination for people who have low vision and/or use a wheelchair.