Data Quality Tools - Why the Cloud is the Cure for Dirty Data

Poor data quality is costing you money. Lots of it. IBM places the cost of dirty data at 3.1 trillion, annually, in the U.S. alone. That’s a staggering amount of loss attributed to incomplete or corrupted data. The good news is that the high cost of dirty data is largely avoidable with the right data quality tools and cloud integration. In this article, we’ll show you why data quality is critical for financial performance and how data quality tools can minimize or eliminate the impact of dirty data on your bottom line.

Understanding Data Quality

Before we discuss data quality tools, let’s stake out what we mean by the term “data quality.” Data quality sounds like something every organization would want, but what does it actually look like? Data quality refers to the usefulness and reliability your data, and whether or not it’s data you can trust. Data quality is determined by the following characteristics:

• Validity — data that provides sound, factually-correct information
• Accuracy — precise data that is error and bug-free
• Consistency — data that performs the same way no matter where its stored or processed
• Relevancy — data that is timely, current, and appropriate for your purposes
• Completeness — comprehensive data with missing values removed
• Accessibility — data that is available for use whenever and wherever you need it

To look at a concrete example, consider the way pharmacies fill prescriptions. When a patient requests a medicine, the pharmacist relies on data integrated from multiple sources to provide the right medicine at the right dosage. This includes patient health records from the prescribing doctor’s office, health insurance information, as well as the pharmacy’s own patient history and drug information. The data from each source must be up-to-date, accessible, reliable, and relevant in order for the patient to receive the correct medication in a timely manner.

Data quality is also critical for integrations of this type. In an integrated system, all departments and locations in the healthcare system should be able to access the same information. If the patient visits another pharmacy or sees another physician, all of this data should be available to these health care providers as well. When data quality is ensured, data moves freely between sources, applications, and destinations.

Data Quality Tools

We know that data quality is essential for efficiency and profitability, but how do we achieve data quality? This can be an especially frustrating problem to solve now that most of us rely on a variety of data formats and sources. Mobile connectivity, the Internet of Things, and the ever-increasing amount of available data will only compound this problem. The solution is a data quality tool.

Data quality tools are programs or applications which analyze datasets in order to identify and resolve problems. A data quality tool automates the steps in this process in order to maximize efficiency and minimize costs. Data quality tools can also be configured to manage data quality for streaming data, data stored in multiple servers, and data that is being prepared for integration.

Data quality tools are compatible with on-premises servers, legacy systems, hybrid, and cloud-native applications. Increasingly, companies are relying on data quality tools hosted in the cloud. This is due primarily to the rise of cloud data storage and the sharp increase in demand for cloud integration solutions. In most cases, data quality tools are delivered through a data integration platform or other service.

How Data Quality Tools Work

Data quality tools are the fastest, most reliable way to deliver data you can trust. In order to understand why this is the case, it’s helpful to look at all the steps required for data quality control. Even if your development team has the skills needed to complete the entire process, it’s likely not the best use of their time. After all, you want your data professionals to focus on innovation, not the routine tasks associated with data quality. Here’s an overview of the way data quality tools work:

Profiling

During data profiling, your data is analyzed to determine its quality, volume, and format. At this point, your data may also be organized or tagged to make it make search and discovery functions more reliable. Metadata will be examined, and overall data quality is assessed.

Matching

Data is examined to identify and merge entries within your dataset. This keeps your data organized and ensures that related values and entries are connected.

Cleansing

During the data cleansing process, duplicate values are eliminated, missing values are completed or discarded, and all categories and field options are standardized.

Enrichment

Existing data is supplemented with other data sources to maximize the value of the data. This includes data integrated from external sources and applications.

Monitoring

Data quality tools can be configured to provide ongoing monitoring of your data. This allows the tool to identify and resolve quality issues quickly, often in real-time, in order to avoid interruptions in data quality.

Data Quality Tools + Cloud Integration for Business Optimization

While it’s helpful to know exactly how data quality tools work, it’s even more important to understand what they can do for your business. Along with cloud integration, data quality tools make it easier to manage multiple data streams, create a single version of the truth, and take advantage of cloud-native applications and analytics tools. Cloud integration provides the pathway, data quality tools make sure the data you deliver has value.

Veolia: Delivering 8,000 Additional Operating Hours Each Year with Data Quality

With 71,000 employees, 637 waste treatment units, and operations stretching across 5 continents, Veolia is the second largest waste management and sanitation company in the world.  With each region and office maintaining their own databases, Veolia struggled to create an integrated database that was efficient, consistent, and reliable.

To resolve this dilemma, Veolia implemented a data integration strategy. This included the use of a data quality tool to ensure that data from all sources was profiled and cleansed before being integrated into central database. As a result, Veolia reduced the cost of developing interfaces threefold and optimized plant availability by adding 8,000 operational hours per year.

Beneva – Using Data Quality Tools to Achieve Customer 360 Goals

Beneva (formerly SSQ Insurance) is the largest mutual insurance company in Canada, offering a wide array of insurance and investment products to three million customers. After 75 years of operation, complex and siloed data systems had made it hard to use customer data effectively. The company knew that it would need to put healthy data at the center of its business. “We weren’t prepared to make any compromises as far as data quality is concerned,” says Simon Latouche, Director of Data Engineering.

Beneva created a unified customer portal which automatically registers customers’ operations. Data Quality and Data Stewardship from Talend ensure high-quality customer data. Now call centers can access more comprehensive data to help customers. Marketers can also run predictive models to customize campaigns. As a result of healthier data, Beneva has tripled its customer win-back conversion rates.

Data Quality Tools and Cloud Integration

The proliferation of cloud-native systems, services, and platforms has made it easier to access data, but has also brought the challenge of unifying and consolidating a wide range of data formats from multiple data streams. The continued growth of mobile and the Internet of Things (IoT) will only compound this problem.

In addition, companies are increasingly seeking ways to integrate data stored on legacy or on-premises systems to the cloud. Cloud integration provides access to a full spectrum of tools for data analysis, processing, and storage. But any integration is only as good as the data being integrated.

Talend Cloud Integration Platform delivers data quality tools to automate and simplify these processes for fast and easy data integrations. Any format, any source. Cloud Integration from Talend also includes advanced security features, 900+ connectors, and a host of data management tools to ensure that your integration runs smoothly from start to finish. Start your free trial today and let data quality be one less thing you have to manage.