Building and managing a modern analytics environment is a challenge facing every data-driven company today. While a large number of companies use cloud-based data warehouses to store and manage their valuable business data, many also rely on data lakes as well as emergent technologies such as data mesh, data fabric, and others.
In a rapidly evolving environment, how can your company decide on the best approach to take when it comes to processing data and gaining insight from it?
Coalesce recently co-sponsored a research project conducted by European analyst firm BARC (Business Application Research Center) in partnership with the Eckerson Group, examining current data warehouse and data vault adoption trends in modern analytics environments.
The survey results revealed what best-in-class companies are doing right—as well as highlighted the mistakes that less successful companies are making.
You can download the full Data Warehouse and Data Vault Adoption Trends report here.
Priorities and pain points
To get a better sense of the on-the-ground reality for data professionals, the survey first asked them to share information about the strategic initiatives their companies were most focused on.
The five top initiatives they reported as being a “high priority” were digital transformation (69%), data modernization (64%), advanced/predictive analytics (48%), cloud migration (42%), and data democratization (30%).
As the report notes, these findings are evidence of the close relationship between digital business and analytics. To compete in today’s economy, companies must digitize customer interactions and operations, modernize their environments, leverage an elastic cloud infrastructure, and use data to generate new insights about the business—while making sure to democratize access to that insight across the organization.
The survey confirmed that few companies are using just a single approach, but rather rely on a patchwork of solutions. Only 18% of companies surveyed rely on a single data architecture; 39% rely on two, 21% on three, and 17% rely on four different architectural types to maintain their data environment. The findings show that the larger the environment, the more likely it is to use multiple types—the data warehouse is no longer relied on as the sole repository of data across a wide swath of the industry.
As for the major pain points these companies feel with their current systems, half (50%) reported “poor data quality” as the top concern, and nearly half (49%) cited “dependence on a few people or consultants.” The other big issues they are facing include “lack of automation” (37%) and “competing shadow systems” (36%). (See Figure 1.)
Figure 1: What are the biggest challenges in your current analytics environment? Source: Data Warehouse and Data Vault Adoption Trends, April 2023 – © BARC GmbH, Eckerson Group 2023.
Digging into data vault adoption trends
One of the main goals of this survey was to gain some real-world insight into how many companies have adopted the data vault approach to data management. Data vault is more than a data modeling technique; rather, it is a hybrid approach that involves people, process, and technology—this is best achieved by following the data vault 2.0 solution developed by the Data Vault Alliance.
The survey revealed that most data vault adopters only partially adhere to the data vault 2.0 solution: only about one third (31%) reported that their approach “fully aligns” with the solution standards, while 60% said it “partially” aligns.
But the survey also discovered that adopting the data vault and making sure to train staff in data vault 2.0 solution techniques is one important key to success, as best-in-class companies do so in much higher numbers than those companies lagging behind. In addition, more of these data leaders (91%) plan to increase the role of the data vault in their environments when compared with data laggards (60%).
Embracing automation
Another key thing that successful companies are doing is expanding the use of automation in their data environment using commercial automation tools (such as Coalesce). Respondents reported that they automate most or all processes for “data integration” (69%), “platform monitoring” (58%), and “data quality monitoring (38%).
As for automation tools, nearly two thirds of best-in-class companies (62%) say their automation is “fully” or “mostly” based on commercial tools rather than homegrown scripts, compared with just 24% for data laggards. (See Figure 2.)
Figure 2: What is the level of automation based on commercial tools vs. homegrown scripts? Source: Data Warehouse and Data Vault Adoption Trends, April 2023 – © BARC GmbH, Eckerson Group 2023.
Ready to turn insight into action?
To read the survey results in detail as well as learn the five main recommendations for companies to derive more business value from their analytics environment, download the full “Data Warehouse and Data Vault Adoption Trends” report.
Ready to take the next step? Learn how you can design and develop your data vault more efficiently with Coalesce for Data Vault.
