Project: #17 Up-to-date city maps for modeling, planning, and assistive technologies Progress Report - Reporting Period Ending: March 31, 2018 Principal Investigator: Christoph Mertz Status: Active Start Date: Jan. 1, 2017 End Date: Aug. 31, 2018 Research Type: Applied Grant Type: Research Grant Program: MAP-21 Grant Cycle: 2017 TSET UTC Progress Report (Last Updated: March 24, 2018, 9:02 a.m.) % Project Completed to Date: 35 % Grant Award Expended: 30 % Match Expended & Document: 30 USDOT Requirements Accomplishments The fields of deep learning and 3D reconstruction from images are rapidly developing. We have been updating our detection and analysis tools to take advantage of new techniques. We have applied Mask-RCNN to traffic sign detection and are now able to detect about 40 different signs. Overhead signs on interstates have text on it and it is desirable to automatically detect and read this text from images. We are testing Foo & Bar method CRNN to detect and recognize text. Their method used quadrangle regression network for text detection, and then used homography to transform quadrangle regions to rectangles and finally CRNN for recognition. Initial tests showed that it is possible to detect and read the text, but not with high accuracy. To improve the system it needs to be trained with with significant number of labeled overhead signs. We intend to do this next. Another research question we are addressing is detecting hard examples, e.g. traffic signs that are defaced or partially occluded. We are testing a special dropout layer in a RCNN to accomplish that task. With regards to localization we tested various visual-odometry packages. The best performing of our purposes was OBR-SLAM2. It works well in some cases, but it fails in others. We believe that the problem is the rolling shutter of the camera. We are investigating methods to deal with that problem. Impacts The startup RoadBotics was founded in 2016 with the technology of earlier work of this project. The company has direct and significant impact on employment, attracting private investment, and improving infrastructure maintenance: Within 15 months of founding, RoadBotics has - Raised $1.2 M in capital, including from multiple out of state institutional investors. About to close an additional +$2M in capital from still more out of state institutional investors. - Acquired 31 paying customers in 10 US states and 2 countries (i.e., US and Australia) with total annual contract value of ~$260K - Created a customer pipeline of $1M in outstanding proposals - Assessed 5,000 lane miles to-date with another 2,500 in the queue scheduled to be collected and assessed - Employed 10 full-time and 8 people part-time professionals as well as 4 full and part-time interns - Secured active channel partnerships with 4 publicly traded multi-billion dollar companies. We’ve also secured 3 regional channel partnerships and are in discussions with 4 large transportation, tech and civil engineering companies for further deployment of our offering. - Piloted a rail and parking lot versions of our product for these two verticals respectively Other The project web page is http://www.cs.cmu.edu/~road/ and a related web site is http://www.cs.cmu.edu/~reconstruction/. New Partners n/a Issues n/a