Data Mesh is a concept that was made famous by Zhamak Dehghani, I first came into contact with it from this article: Data monolith to mesh. She later went on to write a good book on the subject.
The goal for data mesh is to address the complexities of handling large and diverse amounts of data in an enterprise. Data Mesh strives to move away from having centralized data teams that manage the data for all business units and move into a decentralized management of data where the data is provided as data products. Those data products are owned by the different business domains.
The following principles are key concepts in Data Mesh:
- Data as a Product – Data is turned into a product in the same manner as any software product is built to fit business and user needs. The aim for the data products are to be discoverable, addressable, self-describing, have known quality, be secure and interoperable.
- Domain-oriented data ownership – Each domain owns, manages and turns data their data into usable products that are used by themselves and are offered to other domains to use.
- Self-Serve Data Infrastructure as a Platform – Each domain uses the common data infrastructure to build, maintain and provide their Data Products. Each team is fully self-sufficient in handling their data products with minimal dependency on other teams.
- Federated Computational Governance: In order to not have a free-for-all when it comes to how to work with data products and governance of both products and data it is very important to implement proper governance. Therefore Data Mesh aims to establish standardized policies, interoperability mechanisms, and global governance that work across the distributed mesh, while allowing domains flexibility in how they implement them locally.
So, why has Data Mesh become so popular? Because it addresses many of the complexities of working with data in any large enterprise. Especially how to allow freedom of execution within each domain.
Is everything solved in Data Mesh? No, as usual the real world is very complex and the interdependencies between domains can be hard to resolve. However Data Mesh offers several good tools and principles to adopt in a large enterprise.
When to adopt Data Mesh? If you have a large organisation with clear business & data domains and have the maturity to own data products within those domains. It does require quite mature technology thinking in working with data products and with the common data platform.
When not to adopt Data Mesh? If you have a smaller organisation where the data team can support the whole business and the data domains are not as distinct. Implementing data mesh is very hard if the data maturity is low, in that case it may risk becoming a tech endevour rather than a business development.
Do I like Data Mesh? Yes, I do. But as with all methodologies and principles it comes with some pro’s and con’s. I do not see it as a silver bullet but there are really good concepts in there to adopt.
Some further reading:
- Zhamak Dehghani’s original article on the subject: https://martinfowler.com/articles/data-mesh-principles.html
- Data Mesh Resources: https://www.datamesh-architecture.com/
- Zhamak Dehghani’s book: https://www.oreilly.com/library/view/data-mesh/9781492092384/