Abstract
To date, there have been dozens of autonomous shuttle projects across the country aiming to test fully driverless transit operations (Federal Transit Administration 2023). Typically, these pilots involve small shuttles, or “pods,” which carry between 12-15 people and operate at relatively slow speeds (~25 mph). While many of these shuttles operate without incident, there have also been accidents and injuries that have occurred during pilots. For example, in 2017, an automated passenger shuttle in Las Vegas was hit by a truck backing up (Gibbs 2017). Although an attendant was on board, they could not steer the shuttle out of the way because the controls were locked in a compartment (National Transportation Safety Board 2019). More recently, an autonomous shuttle in Orlando collided with a full-length bus at low speed while the shuttle was pulling forward from a stop near the curb and the bus was making a lane change toward the curb (Hope 2023). The onboard attendant attempted to stop the shuttle; however, reports suggest there was insufficient time or space to avoid the collision.
Although it seems counter-intuitive, increased automation can actually make the task of operating a vehicle more challenging. Prior research on human-machine teaming in aviation (Casner and Hutchins 2019) outlines how novel challenges emerge when operators need to take over quickly during emergency situations (Casner and Hutchins 2019) such as skill atrophy (Ebbatson et al. 2010, Pettigrew, Fritschi, and Norman 2018) and mode confusion (Sarter and Woods 1995). As automation takes over more routine aspects of driving, operators are left to manage the most challenging situations. Research on this phenomenon shows that reaction time increases as time disengaged from the task of driving increases, regardless of cognitive engagement (Funkhouser and Drews 2016). Within aviation, automation has reduced many common crash scenarios; however, it has also created new, more complex situations leading to new kinds of crashes (Casner and Hutchins 2019). For example, the tragic Boeing 737 MAX MCAS crashes resulted from a single malfunctioning AOA sensor which provided incorrect data, leading to pilots experiencing sudden and unexpected loss of control authority.
In this research project, we will partner with Beep (https://ridebeep.com/) an autonomous shuttle bus service provider to conduct field observations and interviews with on-board AV shuttle operators and remote operators. Our focus in these observations and interviews will be to understand the current work processes and human factors of the work these operators do. We will conduct cognitive and physical task evaluations of operators’ work processes and usability evaluations of the physical and digital interfaces that they use. Based on these observations and evaluations, we will co-develop recommendations for improved work processes, new training, and potential interface improvements. Overall, we aim to improve the human-machine interaction of on-board and remote shuttle operations, ultimately enhancing the safety of AV shuttle operations.
Description
Timeline
Strategic Description / RD&T
Our project aligns with the US DOT Research, Development and Technology Strategic Plan in several ways: advancing two of its strategic goals and explicitly motivated by one of its primary innovation principles.
Goal 1: Safety — As it stands today, most AV technology has only been proven in near-ideal environmental conditions. However, the world is anything but ideal. Given that transit often operates close to vulnerable road users, it will be critical for automated vehicle perception systems and automated braking to be highly robust. Furthermore, when there is human-in-the-loop operation, there should be effective and clear communication between the automated vehicle systems and the operator.
Goal 2: Equity — Autonomous shuttle vehicles are being explored as a means to connect riders in low density environments, as well as to supplement paratransit operations. Such applications seek to improve transportation accessibility, providing an option for individuals with limited mobility, the elderly, and those without access to private vehicles. Understanding how automation affects the work of operators will be particularly important given the outsized affect safety issues could have on the traditionally marginalized populations who would use these services.
Innovation Principle: Support workers — Prior research in aviation shows that while automation can prevent many common issues and accidents, the added complexity of automated systems creates new kinds of accidents. We are already seeing these kinds of new incidents today such as an incident in Las Vegas where a truck backed into an AV and the trained safety driver could not back out of the way due to having limited controls of the vehicle. Improvements in aviation safety have come from developing an understanding of and designing for human-machine interaction. Without considering the human-machine team, on-board attendant and remote operator work may be intensified, and the real value of automated technologies will not be realized.
Deployment Plan
Phase 1 (July - Dec 2024): Ethnographic Observation and Interviews with Operators of Autonomous Pilot Vehicles
In the first phase of this research, we will employ ethnographic research methods including participant observation and semi-structured interviews to examine autonomous shuttle transportation systems, and the human-automation teams that contribute to their operations. With permission from the autonomous mobility-as-a-service provider Beep (see Letter of Support), we will observe autonomous shuttle operations in several cities across the United States, including the Orlando, Atlanta, and San Francisco areas. With a particular emphasis on the interactions between various stakeholders, the primary objectives of this work include understanding the roles and work of on-board attendants, the tasks undertaken by remote operators, and passenger experiences and perceptions. The study will address several key research questions. First, it will explore how passengers experience shuttle rides, including their general perceptions and how they make sense of the shuttle's performance. Additionally, it will examine passenger interactions with on-board attendants and other passengers, shedding light on their dynamics. Furthermore, the study will analyze how passengers compare Beep shuttles with other transit alternatives (e.g., traditional public transit, personal vehicles).
The research will also delve into the roles and activities of on-board attendants during shuttle rides. This will involve understanding their responsibilities in monitoring and managing the vehicle, as well as their interactions with passengers and non-passengers, such as other road users. The study will further investigate how on-board attendants manage warnings, malfunctions, and other issues while communicating with remote operators, thereby contributing to the safety and efficiency of the service. To gain insights into the essential skills and attributes required for the role, the research will explore the perspectives of on-board attendants and what they view as the most important aspects of their jobs.
In parallel, the study will investigate the activities of remote operators who are responsible for monitoring a fleet of shuttles. This aspect will entail an examination of the specific elements they monitor and attend to, along with the strategies they employ to scan for and identify problems or issues. Additionally, the research will explore the strategies that remote operators employ to intervene and provide support to shuttles in need, ensuring the overall functionality and reliability of the fleet. Similar to on-board attendants, the study will seek to understand the critical skills and attributes that remote operators consider crucial for their role.
This phase of the research holds promise for understanding the current state of automation in urban mobility, contributing valuable insights on the design and operation of autonomous transportation systems, and informing regulatory frameworks. It seeks to shed light on the roles of onboard attendants and remote operators, as well as passenger experiences and preferences, ultimately shaping the future of autonomy within transit while ensuring service quality and safety.
Phase 2 (Jan - Jul 2025): Human Factors Evaluation and Improvements
In the winter and spring, we will build upon our initial observations and interviews to conduct a more thorough human factors evaluation (Stanton et al. 2017) for both on-board operators and remote operators. These will include physical and cognitive task analyses of their work, usability analyses of their interfaces, and process mapping of their tasks to determine where possible issues and errors may occur and to identify possible improvements to their work. In the case of onboard operators, we aim to identify ways to improve the physical ergonomics of their work and to reduce overly taxing cognitive load through improvements to the digital and physical interfaces that they use.
For remote operators, we anticipate focusing primarily on their mental workload and their ability to monitor and manage multiple sources of information. Our work here may also include understanding the remote teleoperation of shuttles and may identify interface improvements for monitoring and control (Kettwich et al. 2021).
We will include additional field observation visits to observe shuttle operations in Jacksonville, FL and Port St. Lucie (North of Miami, FL), as well as observations and more focused human factors interviews in the remote operation center in Orlando, FL.
Following our analysis, we will document our recommendations in a report to be shared with our partners at Beep. The report will include the documentation of the analysis as well as potential process, training, and interface improvements. Process improvements will be documented in diagrams. Training recommendations will be documented in mock-up training guides. Interface improvements will be documented as sketches and medium-fidelity mock-ups created with a digital interface prototyping tool (i.e., Figma).
References
Casner, Stephen M., and Edwin L. Hutchins. "What do we tell the drivers? Toward minimum driver training standards for partially automated cars." Journal of cognitive engineering and decision making 13.2 (2019): 55-66.
Ebbatson, Matt, et al. "The relationship between manual handling performance and recent flying experience in air transport pilots." Ergonomics 53.2 (2010): 268-277.
Federal Transit Administration. “Transit Bus Automation Quarterly Update.” Technical Report Q3 2023 Update. Federal Transit Administration, 2023.
https://www.transit.dot.gov/sites/fta.dot.gov/files/2023-11/Transit-Bus-Automation-Quarterly-Update-Q3-2023.pdf
Fowler, Margaret, Farzan Sasangohar, and Bob Brydia. "Assessing the Development of Operator Trust in Automation: A Longitudinal Study of an Autonomous Campus Shuttle." Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 64. No. 1. Sage CA: Los Angeles, CA: SAGE Publications, 2020.
Funkhouser, Kelly, and Frank Drews. "Reaction times when switching from autonomous to manual driving control: A pilot investigation." Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 60. No. 1. Sage CA: Los Angeles, CA: SAGE Publications, 2016.
Gibbs, Samuel. “Self-driving bus involved in crash less than two hours after Las Vegas launch.” The Guardian, 2017. https://www.theguardian.com/technology/2017/nov/09/self-driving-bus-crashes-two-hours-after-las-vegas-launch-truck-autonomous-vehicle
Hope, Graham. “Autonomous Shuttle Crashes Two Days After Launch.” https://www.iotworldtoday.com/transportation-logistics/autonomous-shuttle-crashes-two-days-after-launch. 2023
Kettwich, Carmen, Andreas Schrank, and Michael Oehl. "Teleoperation of highly automated vehicles in public transport: User-centered design of a human-machine interface for remote-operation and its expert usability evaluation." Multimodal Technologies and Interaction 5.5 (2021): 26.
National Transportation Safety Board. 2019. “Low-Speed Collision Between Truck-Tractor and Autonomous Shuttle, Las Vegas, Nevada, November 8, 2017” Accident Number HWY18FH001
Pettigrew, Simone, Lin Fritschi, and Richard Norman. "The potential implications of autonomous vehicles in and around the workplace." International journal of environmental research and public health 15.9 (2018): 1876.
Sarter, Nadine B., and David D. Woods. "How in the world did we ever get into that mode? Mode error and awareness in supervisory control." Human factors 37.1 (1995): 5-19.
Stanton, Neville A., et al. Human factors methods: a practical guide for engineering and design. CRC Press, 2017.
Expected Outcomes/Impacts
Our human factors evaluations and subsequent recommendations aim to improve on-board and remote operators ability to perform their work safely, comfortably, and efficiently. By reducing physical or cognitive burdens through changes in processes or interfaces, we can help improve the service quality of shuttle bus operations. Our research may also suggest new interface designs that can improve the quality of work. It may also inform new training procedures. In the long term, recommendations and concepts for new interfaces and training may make their way into production systems. The recommendations may also help to inform policy around regulations for controls and interfaces (i.e., steering and propulsion controls, emergency stop controls) inside a shuttle as well as remote operation systems (i.e., information display requirements, error notification systems) or training requirements for onboard and remote shuttle operators.
Expected Outputs
The research seeks to understand the work of onboard and remote shuttle operators, evaluate their physical and cognitive tasks, and make recommendations for improving their work processes, training, and interfaces for interfacing with automated shuttles. We will produce the following outputs:
Two academic papers based on our observations of real-world operations and human-machine teaming work of 1) on-board operators and 2) remote operators. These papers will be published in relevant human-computer interaction and transportation research venues.
Two human factors evaluation reports for on-board and remote operators. These reports will be shared with our partner, Beep.
A set of process, training, and interface design proposals that will make suggestions for improving human-machine teaming work for on-board and remote operators. These proposals will be co-developed with our partner Beep, and may possibly lead to patentable designs.
A white paper translating our academic findings into recommendations for industry practitioners.
We also plan to submit an NSF proposal to seek additional funding for this project. The attached letter of support from Beep denotes this.
TRID
The scholarship most relevant to our proposed work is a recent article from Fowler et al. (2021), “Assessing the Development of Operator Trust in Automation.” The piece describes a 12-week demonstration project on the Texas A&M University campus, conducted through a partnership with NAVYA. The pilot employed 5 student safety drivers (without Class C licenses), who received a 1-week training from NAVYA on how to operate the vehicles. Through analysis of surveys and interviews with operators over the course of the project, the results showed consistent trust levels among operators throughout the deployment, with no significant correlation found between trust and the severity of operational issues.
Drawing from this initial work, we align our research with Walk et al.’s call for future work to examine how stress and workload affect the role of human operators working alongside autonomous vehicle technology (both on-board and remotely). Beyond measuring trust, our work will focus on opportunities for enhanced safety through specialized work processes, new training, and potential interface improvements.
Fowler, Margaret, Farzan Sasangohar, and Bob Brydia. "Assessing the Development of Operator Trust in Automation: A Longitudinal Study of an Autonomous Campus Shuttle." Proceedings of the Human Factors and Ergonomics Society Annual Meeting. Vol. 64. No. 1. Sage CA: Los Angeles, CA: SAGE Publications, 2020.
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| sarahf@andrew.cmu.edu |
Fox, Sarah |
Carnegie Mellon University |
PI |
Faculty - Untenured, Tenure Track |
| nikmart@cmu.edu |
Martelaro, Nikolas |
Carnegie Mellon University |
Co-PI |
Faculty - Untenured, Tenure Track |
Budget
Amount of UTC Funds Awarded
$95000.00
Total Project Budget (from all funding sources)
$200000.00
Documents
Match Sources
No match sources!
Partners
| Name |
Type |
| Beep |
Deployment & Equity Partner Deployment & Equity Partner |
