Project: #342 Training Drivers to Automated Vehicles Progress Report - Reporting Period Ending: Sept. 30, 2022 Principal Investigator: Helen Loeb Status: Active Start Date: July 1, 2020 End Date: June 30, 2023 Research Type: Applied Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - University of Pennsylvania Progress Report (Last Updated: Oct. 4, 2022, 8:50 a.m.) % Project Completed to Date: 75 % Grant Award Expended: 75 % Match Expended & Document: 0 USDOT Requirements Accomplishments This project aims at developing methods and tools to help drivers of all ages and abilities comprehend and trust Automated Vehicle Technology. This project's goal is to develop an Immersive Mixed Reality driving simulator that can provide a safe stationary platform where drivers of all ages and skills can have a first hand experience and learn what to expect before they go on road. Under these goals and through a collaboration between the University of Pennsylvania and Jitsik LLC, two prototypes are being developed: a chair simulator which has a small footprint, and a car simulator where an actual car is retrofitted with equipment to provide a realistic immersive environment. In the last six months much progress has been made on both versions of the simulator. ----------------------------- Chair simulator The CARLA open source simulator for autonomous driving research was used. This project gave ground to a Master Thesis in Robotics. The CARLA simulator, which is supported by Unreal Engine 4 (UE4) does not provide for a Virtual experience. As a result, the research team had to add the OpenXR plugin and rebuild the project to enable rendering on an Oculus Quest. 2 headset. Rendering high-quality VR simulation in real-time was computationally intensive. To maximize performance, the server and client were separated into two machines, which were connected using an ethernet cable and communicated via the TCP port. The result was a smooth simulation for which 3 scenarios were developed: a highway scenario, a rural scenario and a city scenario. A clinical study of 36 people was undertaken. A brief survey first assessed the level of trust in self-driving of participants. Participants then took part in the simulator experiment. They drove both in manual and automated mode. They were the asked to full in a second survey to assess their change of perception. The study showed that exposure indeed strengthens a positive perception of the technology. ----------------------------- Car simulator In the Fall of 2022, the research team moved the driving simulation experience to an actual car. A Toyota Prius was picked. IMU sensors were affixed to the steering wheel and pedals so as to provide wireless input to the simulation. Preliminary results are satisfactory in the sense that we did not observe much latency. Once the car simulator get further developed, we anticipate the development of a second clinical study, this time in the vehicle. This project has provided strong opportunities for professional development. In the past 6 months eight students (2 from the University of Pennsylvania and 6 from Drexel University) have worked on this project in mechanical engineering, instrumentation, control, perception, Virtual Reality scenario design. Results are being disseminated through a Master Thesis in robotics, 1 scientific paper, a presentation at the International Transportation Systems World Congress in LA (September 2022) and outreach material (2 blogs were written by students about their work and posted online). Also, we attended the AAA 2022 Forum on Users and Vehicle Automation in Arizona and shared our research with auto manufacturers. During the next period . we will complete the in-car simulator. A proposal was submitted to UPenn for funding of a clinical study of older people. We are eagerly waiting for the release of the Cambria Mixed Reality Headset which will help us have a color passthrough in the simulator. We are also building a menu that will enable us to launch a variety of driving simulation scenarios from the headset. Impacts Our work is especially suited for technology transfer as it is a collaboration between the University of Pennsylvania and a start-up company Jitsik LLC. The company rents a garage half a mile away from Penn and provides lab space for experimentation on a real car (Toyota Prius). While academic research is conducted by UPenn researchers and students, outreach is conducted as a day to day operation of the startup (e.g. contacts with potential customers such as driving schools or dealerships), dissemination through social media etc, field stake holders (PAVEcampaign...), business contacts and investors. Specific benchmarks: - 2022 Penn SEAS Outstanding Master Thesis Award (Best Thesis Award for Zhijie Qiao) - Video won FIRST PRIZE at the 2022 Intelligent Transportation System World Congress (Xiatao Sun)- - proposal "Symbiotic Learning: interactive Imitation Learning for Safe Shared Autonomy Systems" submitted to Penn as a collaboration between the Penn School of Engineering and the Penn School of Medicine Other Results of the partnership between the University of Pennsylvania and the JITSIK LLC startup are published in scientific media but also social media for outreach. - The website www.jitsik.com leverages a facebook group, an instagram, a twitter account, a linkedin account that are used to disseminate information about the project. Videos can also be posted on a youtube account. https://www.youtube.com/watch?v=S_ylWJXGNkE - Four blogs were written by student from Drexel University and the University of Pennsylvania: https://www.linkedin.com/company/33202653/admin/ Using Unity 3D for solid engineering applications by Allen Wu Using Augmented Reality Technology for driving simulation by Amine Mrad The DriveRight project: HMI for self-driving by Zhijie Qiao Transitioning to a world of autonomous vehicles by Venkata Gurrala Of note, a pre-existing non-provisional patent, property of Jitsik LLC is currently under review. We are planning to develop educational aid through a pilot in Year 2. Outcomes New Partners A background activity of this research project is the presentation to stake holders of all sorts, especially as it relates to the business development of the startup: - contacts for STTR, SBIR development (Clemson University) - contacts for new academic partnerships (Villanova University) - contacts for business (Autoware) - contacts for marketing (SafeDrivingCoach.com, Mike Peretz marketing...) We hope to develop in Year 2 a solid video of prototype that can help further our deployment goals.the second March 2022 update: we will attend AAA Foundation Conference and will look for partners for following years. We are also about to explore deployment in our first driving school. September 2022 update: we are some collaboration with the Perelman School of Medicine (Michelle Johnson, Lisa Walke) as well as a driving school Robert Gillmer. Issues The COVID pandemic has resulted in a slower development as the Jitsik lab could not be used for development at the height of the pandemic. While progress could be made by students who borrowed hardware and worked from home, other development was slowed down such as the development of the car simulator (required the lab). In addition, it became clear as the pandemic extended over a long period of time that clinical trials could not take place in Year 1. We have focused on the engineering development and are currently developing the IRB protocol that will be needed in Year 2 for a pilot of participants. As far as budget is concerned, we are on target. March 2022 update. We are on target for clinical trial and budget. September 2022 Update. Both the mlab and Jitsik have resumed in person activities. We are starting strong for the new academic year with multiple grant submissions, new students, and upcoming demonstrations of our work. We do not anticipate issues at this point.