Data mesh is a popular topic in business right now—especially in the C-Suite, where many executives hope it will be the answer to a host of data problems plaguing companies today. But it’s been challenging to transform data mesh from an intriguing theory into a useful, real-world model. Everybody’s heard of data mesh, but is anyone actually implementing it successfully?
Tackling a growing business problem
The data mesh paradigm was originally developed by Zhamak Dehghani to solve a core problem in the modern business landscape: businesses today are collecting an ever-growing amount of data, and it’s increasingly challenging to leverage those enormous volumes of data for effective business decision-making. Managing data as a core business asset has traditionally fallen on a centralized team—usually IT or a data engineering team—but as the volume of data grows, this centralized IT approach becomes less efficient.
One reason for this is that these teams are always busy reacting to requests from the business or putting out fires, which means they tend to become a bottleneck to other business units that need data for their own analysis and initiatives. These data-hungry teams end up working around the centralized data team to get the data they need, which can produce inconsistent results. Have you ever been in a meeting where you discover that two teams have used different numbers to solve the same problem, even though they’re technically working with the same data sources? This is one of the many symptoms of a centralized IT strategy gone rogue.
At the same time, while data teams have a deep understanding of technology and coding, they may not be as knowledgeable about business strategy and logic. They are therefore not the best group to be tasked with figuring out how data can be applied to make important business decisions.
Data mesh puts forward that the key to solving this problem is decentralization. Rather than relying on a centralized team to manage and supply data to the entire organization, why not give business units (who are domain experts) the ability to build and modify data pipelines? These domain owners are closest to the data that they need, and are best suited to curate it to develop useful data products that can be easily consumed by other business units throughout the organization.
The missing data mesh ingredient
Data mesh is a great proposed solution to real-world data problems—provided you don’t skip the other factors needed to make it successful.
We’ve all heard of these factors before and they’re as relevant as ever. Your people, processes, and technology choices will be what sets your data mesh strategy on the right path forward.
In terms of people, it’s essential that you have support from senior leadership to successfully implement data mesh. A data mesh approach is not something that you can build from the bottom up—you must build it from the top down. And once you have the buy-in from the executive team to operate under this new data mesh paradigm, you need to determine the roles inside each of the business units and establish their participation as well. Who’s going to do what? Who is responsible for each domain?
The process side of things hinges upon governance. Decentralizing data ownership to the organization sounds great—as long as there are data standards that ensure everyone is on the same page in terms of data product quality. Each data product has to be tested, scalable, and discoverable. You will need clear definitions of each and every data element that you publish.
Once you’ve got all this figured out, it’s time to focus on tooling. How exactly are you going to build this? What are the best solutions to support a distributed, domain-driven data architecture? To follow the data mesh model effectively, teams need a place where they can store and process data easily, while making it accessible to every data expert across their organization. Those teams also need self-service capabilities for data discovery, data pipelines, and data analytics. And they need the ability to build things quickly while still observing organization-wide data standards and best practices.
Up until very recently, having the right technology in place has been the key roadblock to the effective implementation of the data mesh paradigm. This is because the technology simply didn’t exist.
The need for a simple yet powerful solution
Because of the different types of data professionals and their different roles and responsibilities in a data mesh implementation, the right technology solution needs to be collaborative and powerful without being rigid. It has to mirror the way data teams work within an organization, granting the autonomy to create data products while maintaining data integrity and strict adherence to data standards. At the same time, it needs to be accessible to data professionals who are less technical, but who require the ability to build the data products they need at the same pace as experienced, technical data experts.
In the past, there weren’t really any solutions that worked for both types of data practitioners. Most solutions skewed toward hardcore coding, making data projects difficult or impossible to work on unless you were a very technical data engineer or architect. Meanwhile, business or data analysts had to resort to last-mile data transformations using business intelligence and data preparation tools because they didn’t normally have advanced technical skills or access to the data warehouse to build data projects or perform data transformations themselves. But these BI and prep solutions are built for business only, and don’t have enough power under the hood to build a strong, scalable data foundation.
Coalesce for data mesh
We built Coalesce to be the best platform on the market for data mesh implementations by making it possible to build and maintain data projects that are accessible, standardized, and scalable for domain teams. Our decentralized, self-service, and governed approach to creating and distributing data products embodies the key principles of data mesh. It’s designed to help data teams treat change as a constant (which it is) and create their own nimble framework for producing data products and maintaining transparency on data heritage, quality, and usage.
This iterative way of developing data products relaxes the tension between overworked centralized data teams and frustrated lines of business, because everyone is getting business- and AI-ready data at the pace that they need it, while borrowing and building on the quality and governed work of others.
As enterprises continue to generate more data—accelerated in part by rapid advances in AI—data mesh will become a prerequisite for effective decision-making and data utilization. Implementing data mesh isn’t without its challenges, but by adopting solutions that are both easy to use and that enable people to build quality data products quickly, companies will be well on their way to reaping the many benefits of a decentralized data ownership strategy.
Ready for a simple yet powerful solution to empower all teams across your organization? Learn more about how Coalesce can help your company successfully adopt a data mesh approach.
