Project: #197 Perception for Transportation Service Robots Progress Report - Reporting Period Ending: Sept. 30, 2019 Principal Investigator: Aaron Steinfeld Status: Active Start Date: Jan. 1, 2018 End Date: Dec. 31, 2019 Research Type: Applied Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: 2018 Traffic21 Progress Report (Last Updated: Sept. 27, 2019, 6:38 a.m.) % Project Completed to Date: 100 % Grant Award Expended: 100 % Match Expended & Document: 100 USDOT Requirements Accomplishments Work prior to this reporting completed our effort on a multi-modal approach to human torso pose estimation and forecasting with a view towards lightweight computational and sensing needs for easy on-board deployment. We have made substantial progress in this area. Our end-to-end system combines RGB images and point cloud information to reason about 3D human pose. We have also demonstrated good performance on predicting torso pose. We extended evaluations from the prior reporting period quantitatively using publicly available datasets (non-transportation settings). As with before, our algorithms continue to outperform complicated recurrent neural network methods, while also being faster on the torso pose forecasting task. A workshop paper on the initial work was accepted and presented in June 2018. A new paper on the latest iteration of this work is in preparation. One of the major barriers to continued development on these kinds of systems is the absence of appropriate datasets for natural human motion in transportation settings. Advances require data sets that have (1) ground truth data on actual human motion, (2) naturalistic behavior in public spaces, and (3) video and sensing from a human/robot height perspective. Data sets exist with two out of the three (mostly #1 and #3), but not of all three. We initially began an effort to collect a modest amount of data from a local site but discovered a demand for this kind of data at a larger scale among other research teams. Therefore, the team partnered with other CMU researchers to collect a large data set with such characteristics. Our team has taken lead on this effort and accomplished two important steps towards the goal. First, the team has obtained permission to collect data in a suitable public building that has the necessary architectural configurations for gathering both ground truth and human/robot views. Second, the team has identified and tested the necessary data capture software and equipment. The hardware is being purchased by a related NSF award on robot social navigation (PI: Steinfeld). The student on this UTC project and this data collection effort are transitioning to the NSF award, which will support the remainder of the data collection effort. The eventual data set will be shared publicly with other researchers, leading to an important resource for people conducting research on human and robot motion in public settings, such as transportation hubs. This project is also associated with the US DOT Accessible Transportation Technologies Research Initiative (ATTRI), specifically through the National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funded Disability Rehabilitation Research Project (DRRP) on Robotics and Automation for Inclusive Transportation. Discoveries and software developed under this award will flow into DRRP out-year efforts. Impacts We have alerted several visiting companies to our data collection effort. They have expressed interest in the new data set and have asked to be notified when it becomes public. Steinfeld has been active in larger US DOT ATTRI outreach efforts. This includes briefing the audience, during the prior reporting period, at the 2018 NIDILRR/USDOT Accessible Transportation Symposium and serving on the "Understanding User Needs, Accessible Design, and Deployment Challenges to Maximize AV Benefits," panel at the 2018 Automated Vehicles Symposium (San Francisco). These presentations were in coordination with US DOT ATTRI personnel. He continues to be in close contact with US DOT ATTRI personnel and to participate in related US DOT efforts on autonomy. For example, he participated in the "Automation and the Workforce Stakeholder Discussion" at US DOT. Steinfeld continues to participate in the Accessibility and Ground Transportation & Parking Working Groups for the Pittsburgh Airport Terminal Modernization effort. Steinfeld also continues to disseminate UTC knowledge to external audiences. For example, he was a panelist for the AV Research Community Virtual Information-Gathering Session, which was co-hosted by the U.S. Department of Labor's Office of Disability Employment Policy and the U.S. Department of Transportation. Other A workshop paper on the initial work was accepted and presented in June 2018. We are preparing a second paper based on this work for peer review. When complete, the new data set on natural human motion in public settings will be a major contribution to the research community. Outcomes New Partners N/A Issues None