JUG Milano Meeting #142

Giovedì 23 Febbraio 2023

Our Journey to the Data Mesh: boosting data in a microservice world

L'incontro si è tenuto Giovedì 23 Febbraio 2023 presso eDreams Via Fara 26 20124 Milano - evento gratuito previa registrazione OBBLIGATORIA (vedi dettagli) (mappa) in Via Gustavo Fara, 26, 20124 Milano MI grazie alla co-organizzazione con eDreams e Red Hat .

La partecipazione in presenza è gratuita e libera, ma è OBBLIGATORIA la registrazione su: form di registrazione per partecipare a JUG Milano in presenza
Prevediamo di effettuare la diretta streaming su YouTube (con VOD a seguire) dell'evento.

Abstract dell'intervento:
What is the best data architecture in the age of microservices and “you-build-it-you-run-it” teams? How should microservices share data between themselves to minimise coupling? How should microservices share data with offline data consumers such as BI, CRM, or ML training pipelines? How can we reconcile data quality, trustworthiness and single-source-of-truth with autonomous teams releasing multiple times per day? Are centralized data governance teams as proposed by data lake architectures compatible with microservices? Data meshes have been proposed as a new data architecture paradigm to answer the questions above, but there are very few public reports on actual implementations and their challenges. We will share the data mesh journey that we started 2 years ago in eDreams, one of the largest online travel agencies in the world. We will focus on three topics: motivation, technical design and transformational approach. First, we explain why we think data meshes are a better fit for microservice architectures. Second, we show the architecture of our current implementation, including how the mesh is connected to the online platform, the technologies we used, and the trade-offs we made for team autonomy, data ownership, quality and accountability, PCI and privacy compliance, self-service data access and reporting, infrastructure cost and budgeting. Third, we share the organizational challenges we faced when replacing the multiple data systems that were working well at the local level. Finally, we will conclude with our current list of open issues and future work.

A cura di Carlos Saona-Vázquez:
Carlos is the Chief Architect at eDreams Odigeo, the largest flight travel agency in Europe, where he has led architectural transformations such as the migration to microservices, the adoption of chaos engineering or the creation of a data mesh. Before eDreams, Carlos served in multiple technical roles in health care, computer security and video game companies designing distributed, scalable and fault-tolerant systems.