Principal Investigator

Bernadine Dias

Status

Completed

Start Date

Jan. 1, 2016

End Date

Dec. 31, 2016

Project Type

Research Applied

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

2016 TSET UTC

Visibility

Public

Abstract

Safety is a primary concern for the visually impaired when navigating unfamiliar urban environments. Since most environments are constructed to be easily navigated by sighted people, visually impaired people have to often seek help and use secondary clues to navigate many urban environments safely. As a result, daily activities such as using transit systems remain challenging tasks for people with visual impairments even though the use of transit systems is often a key factor for participation in employment, and educational, social, and cultural opportunities. 

Visually impaired adults have several challenges when navigating unfamiliar environments.  First, they must pre-plan their navigation routes as much as possible and need to build a mental map of the new environment they will be navigating.  Next, they need to figure out how to navigate between locations of interest from a known environment.  They also need to be informed of dynamic changes to the unfamiliar environment which may impact their safe navigation.  Furthermore, they need to be able to “record” their navigation experience for future trips and also potentially share this information with others who might find it useful.  Finally, if they get into any unsafe or difficult situation while navigating the unfamiliar environment, they need to have a reliable means of getting help.  

We have been exploring specific needs and constraints encountered by this user population when using transit stations.  Based on our findings, we prototyped an accessible smartphone tool that has significant potential to enhance the safety of these travelers. This tool allows travelers to annotate their paths and choose/invite trusted sources to enhance the relevant information that can enhance the safety and efficacy of their travel. The proposed work will further develop this prototype, conduct user tests and iterative enhancements, and deploy the tool with a small set of relevant users in Pittsburgh.
    

Description

Over the past years, funded by the UTC, we have been able to study more closely the needs and challenges of blind and visually impaired (B/VI) people as they navigate urban environments in Pittsburgh; with a focus on transit stations.  Through this work, our findings indicate a strong need for a tool that allows these travelers to receive assistance in rote planning and navigation, annotate routes with their own notes of useful information, and to easily obtain and use relevant information from trusted sources, especially in difficult situations. These trusted sources can fall in the category of authorities, individuals in the area who have been vetted or have a reputation for providing trustworthy information of relevance, or personal contacts (both sighted and B/VI) who the specific user trusts to provide useful and accurate information.  The information needed and the level of detail/nature of the descriptions needed can be very different for people with different types of disability and/or their familiarity with the environment.  In order to further investigate the idea of this tool, we developed an Android smartphone prototype of the proposed tool. While Google Maps provide online map services that enable route planning, it does not provide continuous and dynamic information and notices that are often useful for B/VI travelers to accomplish safe and independent navigation. In addition, it does not allow users to verbally annotate their route or navigate indoor and outdoor environments seamlessly. This is because Google Maps does not provide a real-time map and mainly targets sighted users. As it is often crucial for B/VI travelers to be informed about dynamic changes especially when traveling in unfamiliar environments and to record their observation on the changes for future trips, we developed a framework for incorporating information from trusted sources, and user annotations. 

The underlying concept of the trusted source interface is that trusted individuals can share their observations about dynamic changes in the environment with B/VI travelers for navigating safely and independently. For example, if a street is under renovation and a trusted individual traveling via this route observes this dynamic change that could be a potential risk to B/VI travelers, then he/she can record the dynamic information through the trusted source interface. B/VI travelers who have added that individual to their list of trusted sources are then alerted of this dynamic change, and can choose to avoid the street under renovation and take an alternative path to reach their destination safely. Examples of trusted individuals could be government officers in the locality, property managers (e.g., a building manager), orientation and mobility experts, friends of the B/VI traveler, or B/VI travelers themselves. B/VI users play an important role in this methodology since the system enables them to share their personal navigational experience with other B/VI users. Many B/VI people prefer obtaining navigational information from other B/VI persons due to having the same, or very similar, situational and informational awareness, and because of the types of descriptions and landmarks used in common. 

We developed two interfaces for this prototype tool: 1) for B/VI trusted sources, and 2) for sighted trusted individuals. The B/VI interface of the prototyped tool is made accessible via on-screen gestures, voice commands, and audio output. Locations of interest to the user can be stored as phone contacts and effective routes between destinations (and from the current location) can be calculated via Google Maps. While navigating with this tool, the user is given audible navigational instructions at waypoint intervals, e.g., "Head north for 20 meters and then turn left." In addition, the street name and user’s direction of travel are announced at intersections. The nearby points of interest are also automatically announced to the user for better localization and orientation. If the user deviates from the desired path at a given setting, e.g., 10 meters, the app informs the user to stop and re-routes a new path to the destination. 

Since on-screen gestures are a commonly used input modality for B/VI users when interacting with a touchscreen smartphone, we adopted these gestures as part of our accessible interface. We first conducted a small usability study with a few B/VI users from our partner networks to determine which gestures are more effective for our tool. We also evaluated accessibility and ease of operation of our smartphone tool through this small usability study. For the annotation component, a B/VI user can verbally record his or her navigational experience and refer to it for future trips using our “breadcrumb” interface which allows a user to record messages tied to specific locations on a route that will automatically be played when they encounter those waypoints in the future. Message examples include any potential hazards, a waypoint name, and orientation information for future trips. 

For sighted trusted users, we developed an additional app where they can simply tap the map on the screen and annotate any observed dynamic changes. Trusted users can specify attributes for the data such as 1) characterization of traversability of the waypoint, 2) the proximity of this waypoint to a key landmark, and 3) an estimated lifetime for this data to exist. The users can define a fixed lifetime in hours and minutes or can leave it as an unknown lifetime.

Trusted sources data, either from B/VI users themselves or sighted users, are then directly sent to the local server for storing. The data will be retrieved by the NavPal app depending on the trusted sources selected by B/VI users. The trusted user interface allows B/VI users to designate trusted sources for navigation aids from their contact list. This feature is vital because it enables B/VI travelers to prioritize trusted users and sources and also prevents retrieval of excessive and unhelpful information. This list is stored in the user’s Android internal memory and can be edited through the setting option of this interface. Finally, B/VI users can be informed of dynamic changes by retrieving the user-designated trusted sources.

This initial prototype helped us to develop limited functionality and get some useful user feedback which confirmed the significance of its potential impact in improving safety for B/VI travelers. Over the last year, based on feedback from users and relevant experts, we worked on the following enhancements to the initial prototype, which necessitated a complete re-development of the NavPal app:
•	We have re-designed the interface significantly to enhance usability for blind users
•	Enabled a single app to seamlessly provide navigation guidance in indoor and outdoor environments and detect transitions 

Ongoing work to further enhance this app includes the following:
•	We are exploring the most effective means for including several categories of trusted sources
•	Exploring options in customized route planning
•	Further enhancing the interface for both blind users and sighted trusted sources 
•	Improving options for annotations, and for re-planning routes based on dynamic information
•	Further iterative testing and hardening of the newly developed Android app

Due to the unexpected need for developing a completely new app that can seamlessly provide navigation assistance in both indoor and outdoor environments, our larger testing and deployment plans for this year will not be possible as originally envisioned. Furthermore, we learned that the previously envisioned deployment plan does not make practical sense given current smartphone ownership and transit station use for the user population we can easily access. Therefore, the proposed work will enhance and complete the testing and deployment of the significantly more capable new NavPal app with a new, more realistic deployment plan in the first half of 2016. Our target users have been eager to be able to use an app that can assist their navigation from transit stops to final destinations, and from current locations to relevant transit stops, so this will be the focus of our new deployment plan. 

This work will also continue to dovetail with two other federally funded projects on campus, which increases its potential for impact. These other projects are funding student and staff contributions to this work. One activity of interest to these other projects is how the NavPal app can be used by blind users to enhance their interactions with assistive robots related to urban travel in the future.




Timeline

•	January-March 2016: User testing and iterative enhancement of the technology
•	April-June 2016: Technology deployment and dissemination

Strategic Description / RD&T

Deployment Plan

We plan to deploy the developed tool with 10-20 B/VI users and relevant sighted experts at two locations in the Pittsburgh area and collect their feedback and relevant usage statistics in the first half of 2016. For these deployments, relevant users will be provided an Android phone with the NavPal app, trained to use it, and then asked to use it to navigate from their preferred transit location close to the selected site, to any number of locations on the site, and back to a preferred transit stop.

Expected Outcomes/Impacts

We plan to deploy the developed technology in two relevant locations and gather usage data and feedback for demonstrating and enhancing its effectiveness.  We also plan to publish and present this work in relevant venues. Our metrics for success will be the number of users who participate in the user studies, the number of encountered failures of the app, the percentage of invited users who agree to use the tool at the deployed stage, the usage statistics during the deployment phase, and the dissemination of this work. The ultimate metric for the impact of this tool will be the level of confidence and trust the app earn from the blind users who participate in our studies and deployment.

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
mbdias@ri.cmu.edu	Dias, Bernadine	Robotics Institute	PI	Faculty - Research/Systems

Budget

Amount of UTC Funds Awarded

$48390.00

Total Project Budget (from all funding sources)

$48390.00

Documents

Type	Name	Uploaded
Final Report	Dias_TSETFinalReport.pdf	May 7, 2018, 4:22 a.m.

Match Sources

No match sources!

Partners

No partners!