The San Francisco Municipal Transportation Agency gets riders where they’re going, thanks to Talend, Disy, and geospatial data
By Talend Team
Every day, hundreds of thousands of residents and commuters in San Francisco, California, use the public transportation services of the San Francisco Municipal Transportation Agency (SFMTA). In addition to the city’s buses, subway system, and famous cable cars, the SFMTA manages comprehensive services including bicycle and e-scooter rentals, as well as permits for road closures.
The SFMTA relies on data to make split-second routing decisions to get travelers to their desired destinations quickly and safely. “Many datasets are constantly created around the San Francisco transportation network,” explains Cheong-Tseng Eng, Data Engineering and Data Services Manager at the SFMTA. “Around 21,000 trips per day are generated by e-scooter alone, and 700,000 boardings per day are recorded in other parts of the system.”
The amount of data that needs to be processed is enormous: around 100,000 transactions are carried out every day, and the SFMTA expects that number to rise to 200,000 per day in the near future. A sophisticated data management system underlies everything the SMTA does. All data streams — routes, accidents, bus stops, temporary construction sites, even data from e-scooters — converge to the centralized data lake, where they can be discovered and used for analytics.
Depending on the service, the SFMTA must ensure response times that range from sub-second to 30 seconds. They needed a powerful data management platform that could handle detailed geospatial data. With Talend and the geospatial plugin from Disy, the SFMTA found exactly what they needed.
Getting geo-spatial data in real time, all in one platform
Talend specializes in making data from a variety of sources usable and discoverable in real time. The Disy plugin was an ideal add-on for the SFMTA because it provides ready-to-use geo-operations and direct integration options for common geo-data sources, including PostGIS, Oracle Locator, Spatial, SpatiaLite, ArcGIS Server, Shapefile, and GeoJSON.
This turned out to be the right approach for the SFMTA: areas or lengths can be calculated, geometries can be intersected, and buffers as well as convex hulls or bounding boxes can be formed — directly in the integration and transformation process. A change between systems for spatial data and operations is no longer necessary. Errors are avoided from the outset, resulting in consistently high data quality.
For the SFMTA, the results have been exactly what they needed. “Using GeoSpatial Integration for Talend from Disy, we are able to manage all data, including spatial data, in one single platform,” explains Cheong-Tseng Eng. “We achieve much faster data processes, while reaching a higher data quality level.”
Wassilios Kazakos, Head of Business Development at Disy, agrees: “With the plug-in, SFMTA is able to provide its customers with consistently high, controllable quality of their mobility data with much faster and more efficient processes. GeoSpatial Integration for Talend, together with Talend software, is perfect for all companies that process and use geospatial and location-based data.”