Augmented data management: Data fabric versus data mesh

• Read original article here

Data fabric and data mesh are emerging data management concepts that are meant to address the organizational change and complexities of understanding, governing and working with enterprise data in a hybrid multicloud ecosystem. The good news is that both data architecture concepts are complimentary. But what exactly are a data fabric and data mesh, and how can you use these data management solutions to take advantage of your enterprise data for better decision-making?

Gartner defines a data fabric as “a design concept that serves as an integrated layer of data and connecting processes. A data fabric utilizes continuous analytics over existing, discoverable and inferenced metadata to support the design, deployment and utilization of integrated and reusable datasets across all environments, including hybrid and multicloud platforms.” [1]

The data fabric architectural approach can simplify data access in an organization and facilitate self-service data consumption at scale. This approach breaks down data silos, allowing for new opportunities to shape data governance, data integration, single customer views and trustworthy AI implementations among other common industry use cases.

Read: The first capability of a data fabric is a semantic knowledge data catalog, but what are the other 5 core capabilities of a data fabric?

Since its uniquely metadata-driven, the abstraction layer of a data fabric makes it easier to model, integrate and query any data sources, build data pipelines, and integrate data in real-time. A data fabric also streamlines deriving insights from data through better data observability and data quality by automating manual tasks across data platforms using machine learning. This improves data engineering productivity and time-to-value for data consumers.

 Check out the Data Differentiator to learn more about Data Fabric. 

According to Forrester, “A data mesh is a decentralized sociotechnical approach to share, access and manage analytical data in complex and large-scale environments—within or across organizations using.” [2]

The data mesh architecture is an approach that aligns data sources by business domains, or functions, with data owners. With data ownership decentralization, data owners can create data products for their respective domains, meaning data consumers, both data scientist and business users, can use a combination of these data products for data analytics and data science.

Watch: What is a data fabric, how does it differ from a data mesh, and where do data stores, data lakes and data warehouses fit into the conversation?

The value of the data mesh approach is that it shifts the creation of data products to subject matter experts upstream who know the business domains best compared to relying on data engineers to cleanse and integrate data products downstream.

Furthermore, the data mesh accelerates re-use of data products by enabling a publish-and-subscribe model and leveraging APIs, which makes it easier for data consumers to get the data products they need including reliable updates.

A data fabric and data mesh can co-exist. In fact, there’s three ways a data fabric enables the implementation of a data mesh:

When it comes to data management, a data fabric provides the capabilities needed to implement and take full advantage of a data mesh by automating many of the tasks required to create data products and manage the lifecycle of data products. By using the flexibility of a data fabric foundation, you can implement a data mesh, continuing to take advantage of a use case centric data architecture regardless if your data resides on premises or in the cloud.

Learn more about how you can use a data fabric to put your datasets to work across use cases such as data governance, customer 360 views, data integration, or even trustworthy AI.

1 “Data Fabric Architecture is Key to Modernizing Data Management and Data Integration” Gartner. 11 May 2021.