JUG Milano Meeting #142
Giovedì 23 Febbraio 2023
Our Journey to the Data Mesh: boosting data in a microservice world
La partecipazione in presenza è gratuita e libera, ma è OBBLIGATORIA la registrazione su:
form di registrazione per partecipare a JUG Milano in presenza
Prevediamo di effettuare la diretta streaming su YouTube (con VOD a seguire) dell'evento.
Prevediamo di effettuare la diretta streaming su YouTube (con VOD a seguire) dell'evento.
Abstract dell'intervento:
What is the best data architecture in the age of microservices and “you-build-it-you-run-it” teams? How should microservices share data between themselves to minimise coupling? How should microservices share data with offline data consumers such as BI, CRM, or ML training pipelines? How can we reconcile data quality, trustworthiness and single-source-of-truth with autonomous teams releasing multiple times per day? Are centralized data governance teams as proposed by data lake architectures compatible with microservices? Data meshes have been proposed as a new data architecture paradigm to answer the questions above, but there are very few public reports on actual implementations and their challenges. We will share the data mesh journey that we started 2 years ago in eDreams, one of the largest online travel agencies in the world. We will focus on three topics: motivation, technical design and transformational approach. First, we explain why we think data meshes are a better fit for microservice architectures. Second, we show the architecture of our current implementation, including how the mesh is connected to the online platform, the technologies we used, and the trade-offs we made for team autonomy, data ownership, quality and accountability, PCI and privacy compliance, self-service data access and reporting, infrastructure cost and budgeting. Third, we share the organizational challenges we faced when replacing the multiple data systems that were working well at the local level. Finally, we will conclude with our current list of open issues and future work.
What is the best data architecture in the age of microservices and “you-build-it-you-run-it” teams? How should microservices share data between themselves to minimise coupling? How should microservices share data with offline data consumers such as BI, CRM, or ML training pipelines? How can we reconcile data quality, trustworthiness and single-source-of-truth with autonomous teams releasing multiple times per day? Are centralized data governance teams as proposed by data lake architectures compatible with microservices? Data meshes have been proposed as a new data architecture paradigm to answer the questions above, but there are very few public reports on actual implementations and their challenges. We will share the data mesh journey that we started 2 years ago in eDreams, one of the largest online travel agencies in the world. We will focus on three topics: motivation, technical design and transformational approach. First, we explain why we think data meshes are a better fit for microservice architectures. Second, we show the architecture of our current implementation, including how the mesh is connected to the online platform, the technologies we used, and the trade-offs we made for team autonomy, data ownership, quality and accountability, PCI and privacy compliance, self-service data access and reporting, infrastructure cost and budgeting. Third, we share the organizational challenges we faced when replacing the multiple data systems that were working well at the local level. Finally, we will conclude with our current list of open issues and future work.
A cura di Carlos Saona-Vázquez:
Carlos is the Chief Architect at eDreams Odigeo, the largest flight travel agency in Europe, where he has led architectural transformations such as the migration to microservices, the adoption of chaos engineering or the creation of a data mesh. Before eDreams, Carlos served in multiple technical roles in health care, computer security and video game companies designing distributed, scalable and fault-tolerant systems.
Carlos is the Chief Architect at eDreams Odigeo, the largest flight travel agency in Europe, where he has led architectural transformations such as the migration to microservices, the adoption of chaos engineering or the creation of a data mesh. Before eDreams, Carlos served in multiple technical roles in health care, computer security and video game companies designing distributed, scalable and fault-tolerant systems.