What is Data Profiling?

The health of your data depends on how well you profile it. Data quality assessments have revealed that only about 3% of data meets quality standards. That means poorly managed data costs companies millions of dollars in wasted time, money, and untapped potential.

Healthy data is easily discoverable, understandable, and of value to the people who need to use it; and it’s something every organization should strive for. Data profiling helps your team organize and analyze your data so it can yield its maximum value and give you a clear, competitive advantage in the marketplace. In this article, we explore the process of data profiling and look at the ways it can help you turn raw data into business intelligence and actionable insights.

Basics of data profiling

Data profiling is the process of examining, analyzing, and creating useful summaries of data. The process yields a high-level overview which aids in the discovery of data quality issues, risks, and overall trends. Data profiling produces critical insights into data that companies can then leverage to their advantage.

More specifically, data profiling sifts through data to determine its legitimacy and quality. Analytical algorithms detect dataset characteristics such as mean, minimum, maximum, percentile, and frequency to examine data in minute detail. It then performs analyses to uncover metadata, including frequency distributions, key relationships, foreign key candidates, and functional dependencies. Finally, it uses all of this information to expose how those factors align with your business’s standards and goals.

Data profiling can eliminate costly errors that are common in customer databases. These errors include null values (unknown or missing values), values that shouldn’t be included, values with unusually high or low frequency, values that don’t follow expected patterns, and values outside the normal range.

Learn how data profiling helps reduce data integrity risk.

Four benefits of data profiling

Bad data can cost businesses 30% or more of their revenue. For many companies that means millions of dollars wasted, strategies that must be recalculated, and tarnished reputations. So how do data quality problems arise?

Often the culprit is oversight. Companies can become so busy collecting data and managing operations that the efficacy and quality of data becomes compromised. That could mean lost productivity, missed sales opportunities, and missed chances to improve the bottom line. That’s where a data profiling tool comes in.

Once a data profiling application is engaged, it continually analyzes, cleans, and updates data in order to provide critical insights that are available right from your laptop. Specifically, data profiling provides:

Better data quality and credibility

Once data has been analyzed, the application can help eliminate duplications or anomalies. It can determine useful information that could affect business choices, identify quality problems that exist within an organization’s system, and be used to draw certain conclusions about future health of a company.

Predictive decision making

Profiled information can be used to stop small mistakes from becoming big problems. It can also reveal possible outcomes for new scenarios. Data profiling helps create an accurate snapshot of a company’s health to better inform the decision-making process.

Proactive crisis management

Data profiling can help quickly identify and address problems, often before they arise.

Organized sorting

Most databases interact with a diverse set of data that could include blogs, social media, and other big data markets. Profiling can trace back to the original data source and ensure proper encryption for safety. A data profiler can then analyze those different databases, source applications, or tables, and ensure that the data meets standard statistical measures and specific business rules.

Understanding the relationship between available data, missing data, and required data helps an organization chart its future strategy and determine long-term goals. Access to a data profiling application can streamline these efforts.

Types of data profiling

In general, data profiling applications analyze a database by organizing and collecting information about it. This involves data profiling techniques such as column profiling, cross-column profiling, and cross-table profiling. Almost all of these profiling techniques can be categorized in one of three ways:

• Structure discovery — Structure discovery (or analysis) helps determine whether your data is consistent and formatted correctly. It uses basic statistics to provide information about the validity of data.
• Content discovery — Content discovery focuses on data quality. Data needs to be processed for formatting and standardization, and then properly integrated with existing data in a timely and efficient manner. For example, if a street address or phone number is incorrectly formatted it could mean that certain customers can’t be reached, or a delivery is misplaced.
• Relationship discovery — Relationship discovery identifies connections between different datasets.

Data profiling in action

With the enormous amount of data available today, companies sometimes get overwhelmed by all the information they’ve collected. As a result, they fail to take full advantage of their data, and its value and usefulness diminish. Data profiling organizes and manages big data to unlock its full potential and deliver powerful insights. Talend is helping companies do exactly that.

Domino’s data avalanche

With almost 14,000 locations, Domino’s was already the largest pizza company in the world by 2015. But when the company launched its AnyWare ordering system, it was suddenly faced with an avalanche of data. Users could now place orders through virtually any type of device or app, including smart watches, TVs, car entertainment systems, and social media platforms.

That meant Domino’s had data coming at it from all sides. By putting reliable data profiling to work, Domino’s now collects and analyzes data from all of the company’s point of sales systems in order to streamline analysis and improve data quality. As a result, Domino’s has gained deeper insights into its customer base, enhanced its fraud detection processes, boosted operational efficiency, and increased sales.

Data quality for customer loyalty

Office Depot combines an online presence with continued, offline strategies. Integration of data is crucial, combining information from three channels: the offline catalog, the online website, and customer call centers.

Among other things, Office Depot uses data profiling to perform checks and quality control on data before it is entered into the company’s data lake. Integrated online and offline data results in a complete 360-degree view of customers. It also provides high-quality data to back-office functions throughout the company.

Higher customer lifetime value with healthy data

Globe Telecom provides connectivity services to more than 94.2 million mobile subscribers and 2 million home broadband customers in the Philippines. Opportunities to expand market share are limited, so it was vital that Globe get a better understanding of its existing customer base so it could grow the lifetime value of each relationship.

To deliver the customer insights the business required, Globe needed data that was healthy and suitable for applications such as data analytics. But this proved to be a challenge in areas like data scoring, which at that point was manually addressed by using spreadsheets and offline databases to apply validation and data quality rules to existing data.

Today, Globe operates a center of excellence for its data that encompasses data quality, data engineering, and data governance. Talend provides the company with data scoring, data profiling, and data cleansing capabilities.  With healthy data, Globe improved the availability of data quality scores from once a month to every day, increased trusted email addresses by 400%, and achieved higher ROI per marketing campaign, with metrics including a 30% cost reduction per lead, 13% improvement in conversion rates, and 80% increase in click-through rates.

Data profiling with data lakes and the cloud

As more companies store enormous amounts of data in the cloud, the need for effective data profiling is more important than ever. Cloud-based data lakes already allow companies to store petabytes of data, and the Internet of Things is expanding our capacity for data by collecting vast amounts of information from an ever-evolving range of sources including our homes, what we wear, and the technologies we use.

Staying competitive in the modern marketplace — increasingly driven by cloud-native big data capabilities — means being equipped to harness all that data. From maintaining compliance standards to creating a brand known for outstanding customer service, data profiling is the difference between success and failure when it comes to managing data stores.

Ready, set, profile

Data profiling doesn’t have to be done manually. In fact, the most efficient way to manage the profiling process is to automate it with a data management solution. Data profiling tools increase data integrity by eliminating errors and applying consistency to the data profiling process. Talend Data Fabric’s capabilities allow you to extract, process, and profile data from virtually any source to your data warehouse, without the painstaking process of hand coding.

Start a free trial to find your fastest path to data integration.