Скачать с ютуб Building a Streaming Data Pipeline for Trains Delays Processing в хорошем качестве

Building a Streaming Data Pipeline for Trains Delays Processing

A major cause of dissatisfaction among passengers is the irregularity of train schedules. SNCF (French National Railway Company) has distributed a network of beacons over its 32,000 km of train tracks, triggering a flow of events at each train passage. In this talk, we will present how we built a real-time data processing on these data, to monitor traffic and map the propagation of train delays. During the presentation we will demonstrate how to build an end to end solution, from ingestion to exposure. The presentation will take place as follows: -Data Pipeline: how we set up a data transformation pipeline using Spark 3 and Delta with Azure Databricks and how Delta Lake makes dynamically updated data reliable -Exposure: how we expose our output in the best way depending on the consumer Power BI or REST API. -Production-ready: finally, we will demonstrate how we have structured our development process to make it reliable and aligned with SNCF best practices. About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: https://databricks.com/product/unifie... See all the previous Summit sessions: Connect with us: Website: https://databricks.com Facebook:   / databricksinc   Twitter:   / databricks   LinkedIn:   / databricks   Instagram:   / databricksinc   Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-nam...