Abstract
Two transit buses in different environments will be equipped with cameras and an edge computer connected to a cellular network for time critical and WiFi for high bandwidth communication. The computer detects relevant events and saves data of interest in a privacy preserving manner. This system will enable many mobility relevant observations related to traffic, weather, infrastructure, or disruptions. We will implement several of them, the major one will be landslide prediction. We will partner with RoadBotics to commercialize the system.
Description
Data is essential to maximize mobility and make the transportation system resilient. Examples are real time schedules, traffic conditions, weather conditions, or infrastructure inspection. In this project we want to develop an efficient data collection platform that can monitor complete routes on an hourly basis. The idea is to make use of transit buses that travel on main roads on a regular basis. They are often already equipped with cameras and an electronics cabinet that houses recording equipment and the automatic annunciation system. We will add an edge computer to it, i.e. a computer that will analyze the video streams live to detect relevant events and sends these detections over a cellular network to a central location. Data that is interesting but not time critical will be saved and uploaded via wifi at the end of the day. Potentially relevant data will be saved for a set amount of time locally and analyzed or uploaded upon request. The edge computer will also handle privacy: Before data is saved any personally identifiable information will be eliminated, e.g. by blurring out faces or license plates. The data collected by such a system can be used for event detection, traffic modeling and infrastructure monitoring and thereby provide input for up-to-date detailed maps of roads and traffic. Up-to-date detailed maps are one essential component of autonomous driving, but they are also needed for traffic management, planning, and infrastructure maintenance. Concrete examples are traffic counts, counting of parked cars, observations of road construction, pothole detection, detecting landslide precursors, measuring snow cover, or observing crossing of wildlife.
In this project we will develop and implement the core edge computing, basic detection and privacy modules and analysis tools, and landslide detection and prediction as a major example. We will give other groups the opportunity to run their own algorithms on the edge computer or further analyze the saved data. By the end of the project the system will run on two transit buses in two different environments. Finally, we will explore commercialization of the system. Christoph Mertz will lead the project and three faculty will be co-PIs, each supervising a graduate student and one major part of the project. Two other CMU faculty/research staff want to use the collected data for their own, independently funded research. We will collaborate with Ohio State which has a parallel project. Allegheny County will advise on the landslide part of the project. Also involved will be RoadBotics, a CMU startup [1]. RoadBotics will run some of its software on the edge computer and help make the results available to the end user. RoadBotics will also lead the commercialization effort. In the next paragraphs we will discuss the various parts in detail:
Edge computing (lead: Mahadev Satyanarayanan (Satya), SCS):
The most cost-effective way to analyze data is to do it on the cloud. However, if the data is far from the cloud (“at the edge”), it can be cost-prohibitive or impractical to move all the data because of bandwidth or latency restrictions. Instead, some or all the data needs to be analyzed at the edge and only the relevant results are communicated to the central location. Satya has worked on edge computing for many years and recently developed a prototype where video is collected and analyzed on a moving vehicle and the results are sent to a central location where it is displayed to the end user [2,3]. This current system is running a single application on a powerful computer. The edge computing on the bus needs to be able to run several applications on a smaller computing platform that is suitable for the power and space restrictions on a transit bus, can withstand the vibrations and is reasonably priced. One should also be able to monitor, supervise and update the system remotely in order to make specific requests or install and test new functionalities. A basic set of modules like the privacy software, counting of vehicle and pedestrian etc. will be running all the time. The algorithms of these modules will be developed in the other sub-projects, this sub-project has to build the software infrastructure to run and monitor them. Then there will be an API where the end user can make requests or implement new algorithms. The system will manage the onboard data, discarding irrelevant data, storing data for possible later use and uploading results according to its time-criticality and the available bandwidth. The system at the central location has to manage the various edge computers, monitor their performance, send requests and updates and receive the results. It accumulates these results and further analyzes them and finally makes them available to the end user.
The day-to-day work of this sub-project will be done for 2-years by a graduate student. Half will be paid by this project, the other half with matching funds.
Base analysis modules (lead: Srinivasa Narasimhan, RI):
The base analysis modules are those implemented computer vision algorithms that are needed for pre-processing or are common to several applications. First among the common modules are detection, classification, and tracking of vehicles and pedestrians. One specific tool that will be developed is the automatic vehicle counting that is needed for the Ohio State UTC project. A first version of this tool will work offline and will enable OSU to analyze the video data that they have already collected and will continue to collect from their transit bus partner. Another important module is one that converts the raw video to a privacy preserving video stream. The reason for privacy is that even if it is legal (at least in the US) to collect video data in public, we believe that a system like the one we are building has to respect the privacy of the citizens. This privacy concern has to be built in from the ground up and be an essential feature of the system. It is best to address it at the source of the data so that personal data cannot be intercepted and is never stored anywhere. The privacy functionality will be implemented as a pre-processing step where we will blur out license plates and faces or even the whole vehicle or person in the video before it is saved.
The group of Prof. Narasimhan is already working on a similar project with stationary cameras. The modules will be jointly developed by the two projects. This sub-project will partially support one graduate student.
Landslide prediction (co-leads: Amit Acharya, CEE, and Christoph Mertz, RI):
Land-slides have been a significant problem for South-Western Pennsylvania in the past year. Above average rainfalls have caused many landslides which resulted in significant traffic disruptions and property damage [4]. Prof. Acharya and Dr. Mertz have started an exploratory project to investigate how recent advances in computer vision combined with civil engineering methods can predict and thereby help mitigate landslides. Daily or even hourly observations from transit buses would be ideal to detect precursor events like cracks, accumulation of debris, or flooding. In this sub-project we will pay particular attention to observing cracks in the pavement and relating it to the stability of the supporting hill by doing finite element analysis. Additional input will be the soil type and topography of the area. The crack detection will be based on the commercial grade road damage detection software of RoadBotics, modified to be able to run on the edge computer. We will then develop algorithms that further analyzes detected cracks, if they have the patterns of landslide cracks, how fast they are changing, size, etc. In parallel we will develop finite element analysis models to relate the type, size and speed of change of the cracks to the strains produced by the soil movement. The goal is to have an early warning system that raises a red flag when there are precursor events. The system will also provide visual and other data so that a human expert can confirm the danger and further analyze the area. We will work with Allegheny County to better understand the practical side of the landslide problem and get feedback on our system.
The day-to-day work of this sub-project will be done for 2-years by a graduate student. One year will be paid by this project, the other year with matching funds.
Other applications (contacts: Sean Qian, CEE and Heinz; Ben Schmidt, RoadBotics; Luis Navarro-Serment, RI):
Landslide prediction is only one of many possible applications for the “Bus on the edge” that improve mobility. Several researches have indicated that they are interested in the data collected by our system. Sean Qian wants to use traffic count as input to his traffic models. He also has an ongoing project with Christoph Mertz to detect and count parked cars. The plan is to run this algorithm on the “bus on the edge” to get detailed parking data that in turn can be used to optimize parking regulations. RoadBotics is interested in observing roads regularly over a long period of time to determine how fast cracks, potholes, or other road damages develop. Luis Navarro-Serment has projects involving pedestrian predictions. For his models to work he needs to observe many people in different situations. He wants to use the pedestrian tracks collected by our system to train his models.
All these will be collaborations with self-supported projects where we provide data and basic analysis and they give us feedback and other help. We will be happy to provide our data to other projects.
Additionally, we will use the collected data for class projects and independent studies.
Ohio State Collaboration (lead: Mark McCord):
Mark McCord and his colleagues at OSU have a project funded by their UTC “Using municipal vehicles as sensor platforms to monitor the health and performance of the traffic control system.” They use video from transit buses to obtain traffic flow estimates across extensive urban roadway networks. This is obviously a great application to run on our proposed system and we have both agreed to collaborate. Initially OSU will provide CMU some of their video data collected from their transit bus. With this data CMU can immediately start developing detection and analysis tools. Of particular interest to OSU is an automatic vehicle counting, tracking and classification tool. Currently they are manually counting the vehicles in the video through a Matlab-based GUI. The first version of the proposed tool will be for off-line use where OSU can automatically analyze the data they have already collected. Later we will make an on-line version that runs on the edge computer. The algorithm from OSU that converts counts and tracks to traffic flow will then be added as a second layer.
One option is to equip one transit bus in Columbus for the second year of the project.
Bus systems (co-leads: Christoph Mertz, RI; John Kozar, RI):
We will install the hardware and software on two buses. We will start with one bus in the first year. The CMU-UTC leadership has contacts to several bus companies and we will select the most suitable one. The criteria are the routes the bus is taking (needs to be on routes that are prone to landslides), the equipment that is already on the bus, ease of installation, and the willingness of the transit company to collaborate. Our preliminary selection of the main hardware is an Apollo RoadRunner 4K™ Ultra High De¬finition Mobile Recording System connected to a Neousys Rugged Intel Skylake Computer with Optional GPU. Ideally the transit bus has a mobile recording system already installed and we need to only add the computer. In the second year we will install another system on a second bus. As mentioned above, this could be a transit bus in Columbus, but it could also be a bus in a different environment, e.g. a bus in a rural area.
End user presentation and commercialization (lead: Ben Schmidt, RoadBotics):
One main contribution of RoadBotics is to help in commercialization. From the beginning we will have in mind to deploy the system and RoadBotics will share its experience of fielding its own collection and analysis system. One key part will be the display of the results to the end user. One option will be to use RoadBotics own GIS display which is an interactive website [5]. RoadBotics has also a network of customers, users, partners, and investors and will gauge their interest in either using data from a “bus on the edge” or partnering to deploy a larger pilot.
Intellectual properties (IP):
The various researchers and collaborators will bring IP to the project and develop IP during the project. Some of it will be open source and everyone can use it according to the open source license. For the other IP we will grant each other royalty-free evaluation licenses for the duration of the project. At or before the start of the project we will clarify the details and make a list of the status of the IPs we will use.
Conflict of interest (COI):
The PI Christoph Mertz is also founder and part-time employee of RoadBotics. He will comply with any CMU required COI reporting and processes. In this project he will only work for CMU, all work or decisions from RoadBotics will be made by other employees. Christoph Mertz will also recuse himself from any decisions to open source IP. Before the start of the project we will consult the COI staff at CMU if any other COI measures are needed or recommended.
[1] www.roadbotics.com. Disclosure: the PI is co-founder of the company.
[2] Video of live demo: https://www.youtube.com/watch?v=TToOb2rTNZU
[3] “Distraction-free Waze” accepted to ACM HotMobile '19 (https://drive.google.com/file/d/1UdNzgIg0DQ0tN0eNxTMZIKba4dl2qo9k/view)
[4] Example: Route 30 closure, https://pittsburgh.cbslocal.com/2018/06/27/route-30-reopens-after-landslide/
[5] https://www.roadbotics.com/demo-access
Timeline
Before the start of the project:
- find transit bus partner for first bus
July - Dec. '19:
1. edge computer: finalize design, test in the lab, install on bus, collect raw video data
2. Landslide: Collect images/videos of cracks caused by shifting soil, train current crack detection software with these examples, build first version of finite-element model of road on hillside for one location
3. Applications: Develop off-line vehicle counting
Jan. - June '20:
1. edge computer: test in the field, add functionality, collect lots of raw video data, start collecting per-processed data and results, develop management system
2. Landslide: Modify crack detection software to run on edge computer, collect time sequence of cracks caused by shifting soil, add details to finite-element model like soil type, water table, etc.
3. Applications: Develop and implement on-line pedestrian and vehicle detection, classification, and tracking
July - Dec. '20:
1. edge computer: install system on second bus, add all functionalities (running applications, saving results, managing edge system remotely)
2. Landslide: Collect and select soil-movement cracks from several locations, find threshold for issuing landslide warnings, tune and test final-element model to compare to observed time sequence of cracks.
3. Applications: Develop and implement on-line saving of data in privacy preserving manner
Jan. - June '21:
1. edge computer: test complete system in the field, find and fix bugs, improve performance, evaluate performance
2. Landslide: Explore other landslide indicators (e.g. debris, flooding), quantitative comparison between observed cracks and finite-element model to evaluate performance, subjective evaluation of landslide prediction by users (e.g. Allegheny personnel)
3. Applications: test in the field, find and fix bugs, improve performance, evaluate performance
4. Commercialization: Present system to potential customers, partners and investors, make plan to scale up deployment
5. Finishing of project: Clean and document all software, document hardware configuration, publish open source software in public repositories, write publications, write final report
Strategic Description / RD&T
Deployment Plan
The goal of the project is to have two "bus on the edge" systems deployed in the field, collecting and analyzing video data and reporting results. From the beginning the company RoadBotics will be involved to explore its commercial viability. They believe that a successful system will have commercial potential and could be a next step in the expansion of their own business. RoadBotics will talk to its customer base (currently almost 100 cities, townships, etc.) to gauge the interest in the data provided by the system. Some of their business partners manage transit buses and might be willing to consider a partnership where they manage the data collection and RoadBotics uses and sells the data. Another avenue would be to deploy the system on a different fleet of vehicles, e.g. garbage trucks, which have a much wider network of roads they travel on. The intention is to start a pilot of a significantly scaled up system by the end of the currently proposed project. RoadBotics will use its network of investors and partners to fund such a pilot.
Expected Outcomes/Impacts
2 systems running on two transit buses:
- edge computer powered by bus and connected to video cameras on the bus
- management software running on edge computer and management computer at central location
- at least four applications running live on the edge computer:
1. data collection for landslide prediction
2. vehicle and pedestrian detection, classification and tracking
3. Saving of data in privacy conserving manner
4. Crack detection
- successful running of edge system means:
1. Automatic startup and shutdown when bus starts and finishes operations
2. Running for one month with minimal interventions
3. Sending relevant warnings, information, or data through cellular and WiFi network
4. Remotely managing the system on the bus by monitoring its health, sending requests, and changing functionality
- successful running of system at central location means:
1. Display of relevant results in a timely manner through a user-friendly interface
2. Display of health indicators of the edge system
3. Issue of requests, updates, and change of functionality to edge system
4. Correctly updating map with new information and removing expired or superseded information
Metrics for individual applications:
Up-time, CPU and GPU usage, precision-recall
Expected Outputs
TRID
Individuals Involved
| Email |
Name |
Affiliation |
Role |
Position |
| acharyaamit@cmu.edu |
Acharya, Amit |
CEE |
Co-PI |
Faculty - Tenured |
| cmertz@andrew.cmu.edu |
Mertz, Christoph |
Robotics Institute |
PI |
Faculty - Research/Systems |
| srinivas@cs.cmu.edu |
Narasimhan, Srinivasa |
Robotics Institute |
Co-PI |
Faculty - Tenured |
| satya@cs.cmu.edu |
Satyanarayanan, Mahadev |
SCS |
Co-PI |
Faculty - Tenured |
Budget
Amount of UTC Funds Awarded
$110000.00
Total Project Budget (from all funding sources)
$110000.00
Documents
| Type |
Name |
Uploaded |
| Data Management Plan |
DMP_bus_on_edge_v1.docx |
March 15, 2019, 1:13 p.m. |
| Presentation |
2019_Mertz_Bus_v01.pptx |
March 15, 2019, 1:17 p.m. |
| Progress Report |
282_Progress_Report_2019-09-30 |
Sept. 27, 2019, 12:27 p.m. |
| Progress Report |
282_Progress_Report_2020-03-30 |
March 24, 2020, 11:43 a.m. |
| Publication |
Predictive Analytics in Automobile Industry: A Comparison between Artificial Intelligence and Econometrics |
Nov. 28, 2020, 10:22 a.m. |
| Final Report |
Final_Report_-_282_qw1UkYl.pdf |
Feb. 17, 2021, 4:56 a.m. |
| Presentation |
Photogrammetry_Neural_Networks_Landslides_Mertz_v1.pdf |
March 29, 2021, 12:15 p.m. |
| Presentation |
Bus_on_Edge_CMertz.pdf |
March 29, 2021, 1:52 p.m. |
| Progress Report |
282_Progress_Report_2021-03-31 |
March 29, 2021, 1:59 p.m. |
Match Sources
No match sources!
Partners
| Name |
Type |
| RoadBotics |
Deployment Partner Deployment Partner |
| Allegheny County |
Deployment Partner Deployment Partner |
| The Ohio State University |
Deployment Partner Deployment Partner |
