Principal Investigator

Christoph Mertz

Status

Completed

Start Date

Jan. 1, 2017

End Date

Aug. 31, 2018

Project Type

Research Applied

Grant Program

MAP-21 TSET National (2013 - 2018)

Grant Cycle

2017 TSET UTC

Visibility

Public

Abstract

Digital maps are important for many different aspects of intelligent transportation. They are needed to model traffic patterns, to plan infrastructure upkeep, or for navigation for different traffic participants. These maps not only need to contain roads and all the transportation relevant objects like lane markings and traffic signs, but also their state of repair, compliance with regulations, and suitability for various users. The last point is particular important for people who use wheelchairs, they not only need to know if there is a sidewalk but also if the sidewalk is wide enough and well maintained. Traditional methods to create such maps are manual surveying or surveying vehicles that make use of specialty sensors. These methods are cost prohibitive to keep maps up-to-date.    Our proposed approach is to use inexpensive sensors on a fleet of vehicles that drive on the road for other purposes. We will build on our experience with creating maps of road damage and stop signs. We want to expand our detection to all regulatory traffic signs and lane markings and measure retro-reflectivity of signs and lane markings. We also want to detect damage, vandalism and vegetation overgrowth. We want to pay particular attention to information that is relevant to people with physical or cognitive limitations or disabilities, like state of repair of sidewalks or size and retro-reflectivity of traffic signs that is important for older driver.    

Description

Existing Capabilities and Data
 In the past years we have developed an infrastructure inventory and assessment system. A smartphone is mounted on the windshield of a vehicle (Figure 1, left) and collects images or videos, GPS, acceleration, turn rate (gyro), and other information. The infrastructures in the images are analyzed and the results are shown in a map. In Figure 1 (middle) is an example of cracks detected in an image and the distress score (ratio of cracked region to total region) is displayed for many roads on a Pittsburgh map (Figure 1 right). A second category we have investigated is stop signs. In Figure 2 one can see a map of stop signs found in Pittsburgh and four examples of stop signs with problems: With sticker, graffiti, occluded by vegetation, and displaced. A fifth stop sign is shown that was completely occluded by a tree. We detected the disappearance of the sign by comparing our inventory with an already existing inventory. We have also done some initial investigation into detecting and assessing lane markings. We are currently running or starting to run pilot projects with the City of Pittsburgh, Marshall Township, Cranberry Township, Penn Township, and North Huntingdon Township. The goal is to implement the road distress and stop sign detection and assessment into their respective maintenance operations. Data from large parts of their road networks have been collected in 2015 (see Figure 2 for coverage of Pittsburgh). We expect by the end of our current project to have complete data sets of the city and townships with pavement and stop sign assessment. We should also have initial algorithms developed that detect some other traffic signs.  
Proposed system extensions
We want our system to inventory and assess the most relevant parts of the road infrastructure. These are most important to plan upkeep and improvement or for traffic simulations. We want to extend our system to deal with all regulatory traffic signs (“stop”, “yield”, speed limit, etc.) and lane markings. We want to inventory them, detect damage and disappearance like we already do for stop signs (see Figure 2). For traffic signs we also want to measure their size. Many older signs are smaller than regulation permits and need to be replaced. Additionally we want to measure the retro-reflectivity of traffic signs and lane markings. Size and retro-reflectivity are particular important for older drivers because eye-sight becomes weaker with age. Retro-reflectivity measurements have to be done at night, where the headlights of the host vehicle will be the standard light source and the brightness of the sign or lane in the image will give the returned intensity. There will be several challenges to this method. Since the headlights do not shine light in all directions uniformly, we need to estimate which light intensity arrives at the sign or lane by estimating the location of the sign or lane with respect to the vehicle. We also need to ensure that the headlights of other vehicles or street lamps do not confuse the system. Another extension of our system we want to implement is to use our pavement analysis tools on sidewalks. We want to be able to find cracking and other damage or signs of deterioration. In addition we want to detect if the sidewalk is suitable for wheelchairs. Large cracks, potholes, steps, or vegetation overgrowth are obstacles that are difficult for wheelchairs. A map of such problems tells the city or township about the repairs or upgrades they need to plan for and lets the wheelchair user plan routes to avoid these sidewalks.


We will continue the collaboration with our city and township partners. They will continue to collect data and make it available to us. We will use the data to develop and test our system. We will make the results available to them in formats that they can use in their maintenance and planning software. They will provide us with feedback and suggestions. We will also work with other CMU groups. We intend to make the infrastructure maps available to the CMU Mobility Data Analytics Center (MAC)1. We have a working relationship with the Rehabilitation Engineering Research Center on Accessible Public Transportation (RERCAPT)2 and they will work with us on the sidewalk assessment.  
Project Tasks
1. Traffic signs
a) Develop method to detect all regulatory traffic signs b) Develop method to assess the traffic signs (defacement, occlusion, displacement) c) Develop method to determine size of traffic signs d) Develop method to measure retro-reflectivity of traffic signs.
2. Lane markings
a) Develop method to detect lane markings b) Develop method to assess the lane markings (deterioration) c) Develop method to measure retro-reflectivity of lane markings.
3. Sidewalks
a) Apply pavement assessment to sidewalk b) Develop method to determine suitability of sidewalk for wheelchairs
4. Collaboration
a) Conduct pilot project with city and townships b) MAC and RERCAPT
Future Work
There are many more parts of the road infrastructure that can be seen in our data and we want to eventually inventory and assess all of them. Examples are vegetation, street lamps and man holes. We want to address these in the following years.

Timeline

See attached proposal

Strategic Description / RD&T

Deployment Plan

Our system is already used by the City of Pittsburgh and several townships in pilot tests. Any new capabilities we develop in this project can be used in the pilot projects. We are in negotiation with some companies and individuals to commercialize the technology.

Expected Outcomes/Impacts

The overall goal is to have the technology used by our deployment partners.
Detection metric: Precision/recall, goal: better than 90%/90%
Assessment metric: Precision/recall, goal: better than 50%/90%
Retro-reflectivity metric: Accuracy (% deviation from ground truth), goal: better than 20%
Suitability of sidewalk for wheelchair: Precision/recall of drivability detection. Goal: better than 50%/80%

Expected Outputs

TRID

Individuals Involved

Email	Name	Affiliation	Role	Position
cmertz@andrew.cmu.edu	Mertz, Christoph	Robotics Institute	PI	Faculty - Research/Systems
jinhangw@cs.cmu.edu	Wang, Jinhang	Robotics Institute	Co-PI	Faculty - Research/Systems

Budget

Amount of UTC Funds Awarded

$127500.00

Total Project Budget (from all funding sources)

$127500.00

Documents

Type	Name	Uploaded
Presentation	Smart_phone_based_infrastructure_and_weather_monitoring.pdf	April 18, 2017, 12:54 p.m.
Presentation	Smartphone_based_road_inspection_poster.pdf	April 18, 2017, 12:54 p.m.
Publication	All about the Road: Detecting Road Type, Road Damage, and Road Conditions	Sept. 29, 2017, 4:58 a.m.
Presentation	Smartphone based monitoring of river infrastructure	Sept. 29, 2017, 4:58 a.m.
Presentation	Road Infrastructure Assessment with Smartphones in Vehicles	Sept. 29, 2017, 4:58 a.m.
Progress Report	17_Progress_Report_2017-09-30	Sept. 29, 2017, 7:30 a.m.
Presentation	smart cities and the funding needs and sources for these ventures	March 24, 2018, 9:02 a.m.
Progress Report	17_Progress_Report_2018-03-31	March 24, 2018, 9:02 a.m.
Progress Report	17_Progress_Report_2018-09-30	Oct. 19, 2018, 4:38 a.m.
Presentation	Autonomous_vehicle_technology_final.pdf	Oct. 19, 2018, 6:05 a.m.
Final Report	17_-_Final_Report.pdf	Nov. 21, 2018, 7:28 a.m.

Match Sources

No match sources!

Partners

No partners!