Project: #342 Training Drivers to Automated Vehicles Progress Report - Reporting Period Ending: March 31, 2023 Principal Investigator: Helen Loeb Status: Active Start Date: July 1, 2020 End Date: June 30, 2023 Research Type: Applied Grant Type: Research Grant Program: FAST Act - Mobility National (2016 - 2022) Grant Cycle: Mobility21 - University of Pennsylvania Progress Report (Last Updated: March 31, 2023, 12:47 p.m.) % Project Completed to Date: 75 % Grant Award Expended: 75 % Match Expended & Document: 0 USDOT Requirements Accomplishments This project aims at developing methods and tools to help drivers of all ages and abilities comprehend and trust Automated Vehicle Technology. This project's goal is to develop an Immersive Mixed Reality driving simulator that can provide a safe stationary platform where drivers of all ages and skills can have a first hand experience and learn what to expect before they go on road. Under these goals and through a collaboration between the University of Pennsylvania and Jitsik LLC, two prototypes are being developed: a chair simulator which has a small footprint, and a car simulator where an actual car is retrofitted with equipment to provide a realistic immersive environment. In the last six months much progress has been made on both versions of the simulator. ----------------------------- Chair simulator The CARLA open source simulator for autonomous driving research was used. This project gave ground to a Master Thesis in Robotics. The CARLA simulator, which is supported by Unreal Engine 4 (UE4) does not provide for a Virtual experience. As a result, the research team had to add the OpenXR plugin and rebuild the project to enable rendering on an Oculus Quest. 2 headset. Rendering high-quality VR simulation in real-time was computationally intensive. To maximize performance, the server and client were separated into two machines, which were connected using an ethernet cable and communicated via the TCP port. The result was a smooth simulation for which 3 scenarios were developed: a highway scenario, a rural scenario and a city scenario. A clinical study of 36 people was undertaken. A brief survey first assessed the level of trust in self-driving of participants. Participants then took part in the simulator experiment. They drove both in manual and automated mode. They were the asked to full in a second survey to assess their change of perception. The study showed that exposure indeed strengthens a positive perception of the technology. ----------------------------- Car simulator In the Fall of 2022, the research team moved the driving simulation experience to an actual car. A Toyota Prius was picked. IMU sensors were affixed to the steering wheel and pedals so as to provide wireless input to the simulation. Preliminary results are satisfactory in the sense that we did not observe much latency. Once the car simulator get further developed, we anticipate the development of a second clinical study, this time in the vehicle. This project has provided strong opportunities for professional development. In the past 6 months eight students (2 from the University of Pennsylvania and 6 from Drexel University) have worked on this project in mechanical engineering, instrumentation, control, perception, Virtual Reality scenario design. Results are being disseminated through a Master Thesis in robotics, 1 scientific paper, a presentation at the International Transportation Systems World Congress in LA (September 2022) and outreach material (2 blogs were written by students about their work and posted online). Also, we attended the AAA 2022 Forum on Users and Vehicle Automation in Arizona and shared our research with auto manufacturers. During the next period . we will complete the in-car simulator. A proposal was submitted to UPenn for funding of a clinical study of older people. We are eagerly waiting for the release of the Cambria Mixed Reality Headset which will help us have a color passthrough in the simulator. We are also building a menu that will enable us to launch a variety of driving simulation scenarios from the headset. March 2023 update. The next section is the final update for the project A lot of effort has been made on developing our car simulator so it can be easily deployed in as many settings as possible. Our car simulator only requires 3 sensors and a headset. No computer or laptop is needed. We are designing the simulator so that it can be installed in any car, working or not. Our prototype was completed in February 2023. Sensors were developed so they can communicate directly with the Meta Quest Headset. Our simulator can be installed in any car in less than 5 minutes. The effort has then turned to software development. We focused our energy on the User Interface. The following software features were added: - Virtual Gear - the simulator features a Virtual Gear Shift that can be activated to place the simulator in Park, Neutral, Drive or Reverse - Head Control - since the driver's hands are onn the steering wheel, we developed a user interface that can be activated by simply looking at options in the user menu in the headser - Steering Control - we developed a 3D printed steering wheel that can clip to the steering wheel of any car. While the Toyota Prius does not lock its steering wheel when the engine is off, most cars do lock the steering wheel. Through the design of that clip-on steering wheel, we anticipate any car will be able to be retroffitted into a driving simulator. We provided 6 onths internships to 3 new students this past semester: Quinn Williams, Nathalia Gomez, Banjamin Naab all from Drexel University. We also provided work opportunity to Sean Mac Colgan, also from Drexel University Impacts Our work is especially suited for technology transfer as it is a collaboration between the University of Pennsylvania and a start-up company Jitsik LLC. The company rents a garage half a mile away from Penn and provides lab space for experimentation on a real car (Toyota Prius). While academic research is conducted by UPenn researchers and students, outreach is conducted as a day to day operation of the startup (e.g. contacts with potential customers such as driving schools or dealerships), dissemination through social media etc, field stake holders (PAVEcampaign...), business contacts and investors. Specific benchmarks: - 2022 Penn SEAS Outstanding Master Thesis Award (Best Thesis Award for Zhijie Qiao) - Video won FIRST PRIZE at the 2022 Intelligent Transportation System World Congress (Xiatao Sun)- - proposal "Symbiotic Learning: interactive Imitation Learning for Safe Shared Autonomy Systems" submitted to Penn as a collaboration between the Penn School of Engineering and the Penn School of Medicine March 2023 update - one proposal received funding from the University of Pennsylvania "Shared Autonomy - URF collaborative project". This project which is a collaboration with the Perelman School of Medicine includes Lisa Walke in the team. She is a geriatrician and will help steer a new research project towards the use of Automated Vehicles by older people - a VitalPrize proposal was submitted to NSF and the Bill and Melinda Gates Foundation for Equity in Drivers Education Other Results of the partnership between the University of Pennsylvania and the JITSIK LLC startup are published in scientific media but also social media for outreach. - The website www.jitsik.com leverages a facebook group, an instagram, a twitter account, a linkedin account that are used to disseminate information about the project. Videos can also be posted on a youtube account. https://www.youtube.com/watch?v=S_ylWJXGNkE - Four blogs were written by student from Drexel University and the University of Pennsylvania: https://www.linkedin.com/company/33202653/admin/ Using Unity 3D for solid engineering applications by Allen Wu Using Augmented Reality Technology for driving simulation by Amine Mrad The DriveRight project: HMI for self-driving by Zhijie Qiao Transitioning to a world of autonomous vehicles by Venkata Gurrala Of note, a pre-existing non-provisional patent, property of Jitsik LLC is currently under review. We are planning to develop educational aid through a pilot in Year 2. March 2023 update The project continues to provide an academic setting for students, both from Drexel University and the University of Pennsylvania. In addition to our 3 papers for this last reporting period, students published 2 blogs: Benjamin Naab's blog: https://www.linkedin.com/pulse/jitsik-mid-coop-blog-jitsik-2e?trk=organization_guest_main-feed-card_feed-article-content Quinn Williams' blog: https://www.linkedin.com/pulse/jitsik-mid-coop-blog-jitsik-1e?trk=organization_guest_main-feed-card_feed-article-content Nathalia Gomez's blog: https://www.linkedin.com/pulse/jitsik-mid-coop-blog-jitsik?trk=organization_guest_main-feed-card_feed-article-content In addition, we regularily update our youtube channel with engineering video which document our progress. These are available on the Jitsik youtube channel -METADRIVE OCULUS MR with Jitsik sensors -METADRIVE Mixed Reality in Quest -OMR Car: Passthrough Surface styling and overlay -car interior rear windows Since last reporting, we developed innovative technology for Mixed Reality use in a car setting: an overlay of predefined share that acts as a mask to block the color passthrough. We also developed IMU sensors which communicate directly with the Meta Quest headset through wifi. Last but not least, we developed a 3D printed steering wheel which can clip on any car steering wheel. Outcomes New Partners A background activity of this research project is the presentation to stake holders of all sorts, especially as it relates to the business development of the startup: - contacts for STTR, SBIR development (Clemson University) - contacts for new academic partnerships (Villanova University) - contacts for business (Autoware) - contacts for marketing (SafeDrivingCoach.com, Mike Peretz marketing...) We hope to develop in Year 2 a solid video of prototype that can help further our deployment goals.the second March 2022 update: we will attend AAA Foundation Conference and will look for partners for following years. We are also about to explore deployment in our first driving school. September 2022 update: we are some collaboration with the Perelman School of Medicine (Michelle Johnson, Lisa Walke) as well as a driving school Robert Gillmer. March 2023 update: we received funding from the University of Pennsylvania through URF funds. The new project will include Michelle Johnson from Rehab Robotics from the University City of Pennsylvania and Lisa Walke, MD, physician from the Perelman School of Medicine. As we developed our NSF Vitalprize proposal, we developed new partnerships with: - Chad Forry, Safety/CDL/Driver Education Teacher, DRIVE Director from the Notheastern High School in Manchester PA - Rebecca Yating Liu, from the Whartin School of Business - Madeline Griffith, from the Children Hospital of Philadelphia - Sean McColgan, from Drexel University Issues The COVID pandemic has resulted in a slower development as the Jitsik lab could not be used for development at the height of the pandemic. While progress could be made by students who borrowed hardware and worked from home, other development was slowed down such as the development of the car simulator (required the lab). In addition, it became clear as the pandemic extended over a long period of time that clinical trials could not take place in Year 1. We have focused on the engineering development and are currently developing the IRB protocol that will be needed in Year 2 for a pilot of participants. As far as budget is concerned, we are on target. March 2022 update. We are on target for clinical trial and budget. September 2022 Update. Both the mlab and Jitsik have resumed in person activities. We are starting strong for the new academic year with multiple grant submissions, new students, and upcoming demonstrations of our work. We do not anticipate issues at this point. March 2023 update: We have a solid platform which allows us to use any car to demonstrate our driving simulator. Our focus is now turned to software development for scenario development. This is complex task which will likely require us to build partnerships with existing developers. We hope to re-use driving simulation software ythat has already been developed for Machine Leaning training of self-driving vehicles. We don't anticipate major issue else than development time. Additional resources would speed up the software development so we can effectively make our driving simulator ubiquitous.