New Talend Research Identifies Cloud Data Warehouse Trends and Best Practices for 2019
TDWI survey highlights how to optimize the return of cloud data warehouse investments
Redwood City, CA - February 20, 2019 - According to a new survey conducted by TDWI on behalf of Talend, a global leader in cloud data integration solutions, new cloud data warehouses (CDW) offer broader data capabilities, stronger performance, and greater flexibility than traditional on-premise databases. However, the survey found that while CDWs are often an important first step in digital transformation, enterprises need to follow some best practices to overcome implementation challenges and increase investment return. The full results of the survey with feedback from more than 200 data architects, and senior IT and analytics leaders can be downloaded here.
“TDWI sees a wide range of data-driven IT systems moving to the cloud aggressively, and this includes the data warehouse,” said Philip Russom, Senior Research Director, Data Management at TDWI. “Cloud gives the data warehouse the elastic scale, agnostic storage, multi-tenant access, and controlled cost it needs for modern requirements. However, cloud data warehouses should be complemented with substantial data integration infrastructure to unify the many pieces of the warehouse with all the data sources and targets available.”
Decision Resources Group (DRG) is one example of this. As a company that manages comprehensive data repositories covering 90+ percent of the U.S. healthcare system, DRG was struggling to combine and organize their disparate data sources. Healthcare data is stored in a structured way, creating the need for DRG to clean and normalize millions of records and group data to assess patient needs and market conditions. DRG was successful in switching to a cloud-first strategy by implementing Talend and the Snowflake cloud data warehouse as the foundation of its new Real World Data Platform. Because of this, DRG became 150 percent more productive without increasing costs, and have since onboarded 100 terabytes of data in just three months. The organization can now supply more meaningful data enabling physicians to understand different patient populations and provider markets to interact with them in a more optimized fashion.
While survey respondents noted that adopting CDWs was critical to helping them achieve faster performance and lower costs, and take advantage of cloud features, there were a number of challenges associated with CDWs as well. Over 50 percent respondents indicated “data governance” as a top challenge, closely followed by “integrating data across multiple sources” at over 40 percent, and “getting data into the warehouse” at about 38 percent. Organizations data analytics needs in a CDW are becoming increasingly complex. Over 35 percent respondents expressed the need for in-memory processing, supporting structured and unstructured data, and integration with third-party analytics tools. As a result, CDWs need to accommodate a wide variety of data and serve a broad range of technical use cases.
Interestingly, 62 percent of respondents in the process of implementing CDWs want them to complement a data lake for analytics. All survey respondents were interested in features such as data quality, metadata management, processing and transforming data both before and after data is loaded to a CDW. As these requirements cannot be met solely by CDW technologies, the response suggests a need for integration solutions to complement the infrastructure. CDWs have to be enabled to accommodate a range of use cases, from business to technical, and support increases in speed and scale, while handling both current and future needs.
“Cloud data warehouses enable enterprises to build data-fueled use-case applications and dynamically deploy data clusters. However, to be successful, enterprises need to understand the entire organization’s requirements,” said Ashley Stirrup, chief marketing officer at Talend. “Most enterprises using CDWs expect to be able to push data in and the rest will follow suit. In order to realize the full benefits – outside of just lowering costs – data architects need to be aware of how every business unit is using data and focus on integration.”
Don’t overlook other architecture and data integration needs when building a CDW. Unlike some point solutions, Talend integration helps optimize and scale data processing before and after its placed in a CDW for complex and advanced processing and transformations. Visit Talend to download the full survey results.
Like this story? Tweet: New survey by @TDWI and @Talend uncovers ways to be successful with a #CloudDataWarehouse https://bit.ly/2TDrJHl