Data Quality Metrics to Know and Measure - DZone Big Data

• Read original article here

The most important characteristic for any organization is not data; it is quality data. In a study, IBM estimated that bad data costs the U.S. economy $3.1 trillion per year. Such costs are incurred when your employees spend time cleaning data or rectifying the errors caused by bad data. Apart from financial costs, bad data becomes a source of dissatisfaction and discontent between you and your customers, partners, and other business relationships. 

This clearly explains the importance of housing quality data in your organization. But what exactly is it, and how can you measure data quality? Let’s take a look. 

Data quality means something different for every organization – since everyone uses data for various reasons. You can say that good data quality means that data is fit for use for any intended purpose. 

In the realm of business intelligence, data is used to draw predictions and make critical decisions. This is only possible through good data quality, ensuring the data in use is accurate, correct, and reliable.  

Since data quality is a subjective concept, business leaders often wonder how they can measure if their data quality is good enough. The answer is to use data quality dimensions.  

Data quality dimensions are a list of metrics that help you to measure data fitness for any intended use. The selection of these dimensions depends on the purpose data serves at your company. But commonly, six data quality dimensions are used. Let’s take a look at what they are.    

To ensure data quality, your data should always be correct and depict reality. There are a lot of reasons why the values in your dataset may not be accurate, but there are two most common reasons for this issue: 

Data is captured at multiple sources, such as local files, relational databases, cloud storage, and other third-party applications. During the integration process, data is subjected to various data transformation techniques, including data profiling, cleansing, and standardization. Sometimes, these operations change the real data, and it no longer depicts the actual values. 

A company gathers data from multiple sources, either the owners of data themselves (such as customers) or third-party vendors (selling customer data). Oftentimes, these sources of information do not share a hundred percent verified information, leading you to store data that is not true or correct.  

Completeness relates to the presence of all necessary data attributes. Before capturing data, make sure you have identified the data required to fulfill your organizational business operations. Afterward, make sure that the required data is being captured and entered into your systems appropriately. 

Incomplete data mostly stems from an insufficient analysis of your data requirements. Companies usually do not realize what data they need, and so they end up introducing required attributes later in their data lifecycle, which causes a lot of data records to be left empty and incomplete.  

Consistency ensures that the same data values are present for the same records at different data stores. Organizations commonly use a huge number of data management applications while managing different types of data, such as employees, customers, and finances. A survey showed that an enterprise uses about 123 SaaS applications. Enterprise data is often stored and utilized from these scattered sources. If these disparate sources represent varying forms of the same information, it will lead your teams to work on conflicting information. Data quality – to some extent – is measured by how agreeable data is across various sources. 

It is not only important what data is captured – but how it is captured as well. Data validity means storing data in the right type and format, as well as within an acceptable range. This also includes enforcing data values to follow the required pattern. 

For example, for email addresses, the values should always contain an ‘@’ symbol; for dates, then it should follow a specified date format; and so on. It is important for data to conform to these standards so that it can be deemed valid. Invalid data – even if accurate or complete – cannot be used for any intended purpose.  

Your data is only valuable if it is relevant. The older it is, the less relevant it becomes. Currency refers to up-to-date/current data. Data ages very fast, and certain events can outdate your data (whether it is an employee changing their residential address, social profile, or last name due to a change in marital status). Currency can also be affected by any lag that may be present between an event occurrence and the time it takes for that data to be available for use. If your data integration framework is complex and time-consuming, it could be that your current snapshots of data are weeks or even months old, leading you to present and base critical decisions on outdated information.  

Data quality can also be measured in terms of data uniqueness, which means that there are no duplicate records present in your datasets for the same entity. To ensure data uniqueness, it is important to capture uniquely identifying attributes for each entity. This will help you to only store a new record when the unique identifier does not exist in the database.  

But sometimes, data deduplication can be more complex. For example, healthcare organizations often hide personally identifiable information (PII) from their datasets to protect patient confidentiality. In such cases, you may need to perform advanced fuzzy matching techniques that select a number of attributes and compute the likelihood of two records being similar. 

Organizations are becoming more data-reliant, yet critical decisions based on poor-quality data can lead you to make incorrect and flawed decisions. These six data quality dimensions are a great place to start as they will help you to assess the current state of your data quality and identify what you can do to make your data more accurate, complete, consistent, valid, current, and unique.