Data Integrity Framework: Trends and Best Practices for Success

• Read original article here

This is part of Solutions Review’s Premium Content Series, a collection of contributed columns written by industry experts in maturing software categories. In this submission, Precisely COO Eric Yau offers a data integrity framework, along with trends to be aware of and best practices to consider.

Organizations are increasingly viewing data as a strategic asset as they seek to compete in a digital-first world – and are also aware that true business intelligence can only come from a foundation of trusted data. Unfortunately, research suggests that there is growing gap between the drive for more data-driven decision-making and the levels of confidence that businesses have in the data available to them.

A recent IDC Spotlight paper1 on “Improving Data Integrity and Trust through Transparency and Enrichment” showed that while 63% of data practitioners believe they are expected to make data-driven decisions, only 27% completely trust the data they are working with. This illustrates an urgent need for businesses to bridge a data trust gap between the expectations placed on the use of data and the ability to trust that data to make confident decisions.

As data moves from its source it degrades – which contributes to the data trust gap. As data moves, its context, meaning and shape become distorted. Siloed systems and an inability to effectively enrich data can widen that gap.

Data integrity holds the key to ensuring data trust, combining data integration, data governance and quality, location intelligence and data enrichment to ensure a foundation of data that has accuracy, consistency and context. Below, I’ll outline trends that exist in the market today that are driving the need for data integrity, as well as the four main steps that businesses should consider when implementing their own data integrity strategy.

The IDC Spotlight also outlined several key trends in the market that are increasingly driving the push towards greater data integrity:

Artificial intelligence: Machines are learning from data. If the data is incorrect, invalid or incomplete, it won’t have integrity, and those machines will get a bad education. Without data that can be trusted, algorithms won’t be able to detect anomalies, infer relationships between entities, and automate processes and decision-making.

· DataOps: Organizations that have implemented DataOps have seen significant improvements in their ability to deliver data projects on time. These initiatives require data integrity technologies and methods to support continuous improvement, testing, and deployment of data to the business.

· Cloud Migration: Organizations desiring to take advantage of cloud scalability and elasticity continue to migrate workloads to the cloud. To address the challenges of integrating siloed and complex data to the cloud, ensure that clean data is fed to the cloud, and make decisions about what to move based on regional regulations, data integrity must play a key role. In addition, centralized intelligence about what data is where, across hybrid and multi-cloud environments, offers organizations the ability to exercise greater control over data assets.

· ESG: Data, and transparency into data, is at the core of ESG initiatives. In addition, needing accurate and complete data to report on environmental and social metrics, an organization must be transparent about how it collects data to ensure it is ethically sourced, and it must

ensure that no biases that are built into its analytics. Accurate, complete, and contextualized data is a must to meet the ESG requirements of customers, the market, and investors.

Prioritizing a data integrity strategy will be crucial for organizations looking to respond to these trending topics.

The four key steps to data integrity success

But where should businesses even start when it comes to building their own strategy for data integrity? Below is an overview of the key steps that organizations should consider when establishing a framework for building trusted data:

Step 1 – Data Integration: Building a holistic view of an organization’s data requires tying multiple systems together through mapping and translation. Integration of data across the enterprise, whether in mainframes, relational databases, or enterprise data warehouses, requires a carefully considered approach to bringing the data together under one roof, and in a way that is most aligned to the organization’s strategic goals.

· Step 2 – Data Governance and Quality: A sound data integrity strategy must help the organization understand the lineage of its data, how it is used by the business, and the controls in place around it. And it must be capable of managing and validating data across multiple systems, proactively identifying anomalies or discrepancies, and triggering workflows and processes to correct those errors.

· Step 3 – Location Intelligence: Virtually every data point in the world can be associated with location in one way or another. Location intelligence involves the use of geospatial analysis and visualization to better understand the relationships in your data and use that insight to reduce risk, better understand customer behavior, and increase efficiencies.

· Step 4 – Data Enrichment: Data enrichment adds necessary context to your data to enable data-driven strategies. Whether you’re interested in standard data or dynamic data (like demographic movement and weather changes over time), you can enrich your enterprise data with relevant context.

Ultimately, different organizations will be at different stages on their journey to data integrity, driven by their own unique mix of business needs and priorities. Some might start by integrating data from siloed systems into a cloud data warehouse. Others may need to first adopt data governance to find and understand their critical data. Or they may first enrich their data to fill gaps and add more context.

No matter what the driver may be, it’s important to ensure that robust data foundations are being put in place to support the success of business initiatives. By building a meaningful data integrity strategy around data integration, data governance and quality, location intelligence, and data enrichment, organizations can be confident that they are making smarter business decisions based on data they can trust.