Data Integration in an AWS Environment

IaaS (infrastructure as a service) solutions — of which one of the most well known is Amazon Web Services (AWS) — are an increasingly popular choice for companies and organizations who want to simplify their data architectures and control costs. The reason for their popularity is simple: IaaS allows companies to purchase only the amount of compute resources, data storage, and networking that they need from a host provider.

AWS now accounts for 40% of the global IaaS market, and is used by companies and organizations in every sector. But a common barrier for those wanting to migrate to AWS platform is figuring out how to manage the complexities of data integration processes. With the right information and tools, anyone can take on an AWS data integration project.

In this article, we explore the fundamentals of ETL and data integration in an AWS environment and look at the factors you’ll want to consider when planning your AWS integration strategy.

What is AWS?

In 2006, Amazon Web Services (AWS) launched two flagship products: Simple Storage Service (S3) and Elastic Compute Cloud (EC2). Since then, AWS has increased the scope, depth, and number of its products to become a massive cloud platform which specializes in providing Infrastructure-as-a-Service (IaaS) to its enterprise customers. According to a report by the Synergy Research Group, AWS currently holds a 40% share of the global IaaS market.

The AWS platform provides a wide range of products including security, analytics, and developer tools. AWS also offers more specialized services including game development, virtual reality, and machine learning. As a result of this broad platform, more and more companies are choosing to integrate with AWS. The question for many is not “if” but “how.” The first step in creating an AWS integration strategy is to understand how the process works and what you’ll need to get things underway.

ETL with AWS

One common data integration process is ETL (extract, transform, load). This pulls data from its source, configures it into a usable format, and then delivers it to a target destination. This configuring of the data — known as the data transforming process — involves sorting, filtering, aggregating, mapping, cleansing, and enriching the data so that it is ready for use as soon as it is delivered to its destination.

There are different strategies and tools for executing ETL with AWS. Developers can fully automate some, others require manual inputs, and still others combine automated and manual processes. Each method varies with regard to its ease of use, time to completion, replicability, and the complexity of the data it can manage. This is especially true of the transformation phase of ETL, in which some methods or tools rely on the painstaking process of hand-coding.

When it comes to identifying the right ETL tools for integrations with AWS, two considerations are critical:

• Your ETL tool must have the capacity to read the schema of the source database, catalog the data, and automatically prepare queries to transform data into the AWS data warehouse. 
• Your tool must also be able to create, configure, and run automated ETL jobs. (This is important because ETL processes are often not a single, isolated event. It’s therefore critical to use an ETL tool than can provide continuous integration with AWS and/or create reusable code to avoid having to start from scratch each time you need to run an ETL job.)

Integration tools

Data integration isn’t simply about migrating data from one database to another. It’s also the process that enables workflows to be streamlined and configures communications between systems and components. Ultimately, it’s the complete integration process — not just data migration — that allows you to extract the maximum value from your data.In addition to handling your data migration, data integration tools allow you to:

• integrate workflows across multiple systems into AWS
• make the underlying integration workflows reusable and easily accessible
• provide easy scheduling and orchestration of jobs
• create of a single version of truth

For most companies and organizations, a holistic, cloud-based data integration solution is the most efficient and cost-effective alternative. This approach seamlessly integrates AWS with your existing data roadmap and provides all the tools needed for additional tasks including cloud analytics, data quality, and real-time streaming. And with a platform that manages all of these tasks, you’ll be simplifying the work for your developers and creating value for your company.

Your AWS data warehouse — what to expect

Now that we’ve taken a look at the basics of data integration with AWS, let’s dive deeper into some of the reasons why AWS has become so important to the IT landscape. A broad ecosystem and wide-ranging capabilities make AWS a compelling choice for many companies and organizations, but it’s the real-world functionality that makes the case for most AWS integrations. To demonstrate why integration with AWS is a top priority for many companies, it’s helpful to take a closer look at two scenarios that demonstrate some benefits of AWS integration.

Elastic deployments for improved efficiency

Many companies rely on local on-site servers to deliver data updates to EMR and RedShift clusters in the cloud. In order to make sure that the data can be delivered whenever the updates are ready, these clusters are allowed to run continually, expending energy and incurring costs even during idle time.

An alternative approach activates the clusters only when they are needed. Using a data integration platform to connect with AWS, start and stop functions can be configured to accommodate a single job or manage recurring jobs that automatically run at specified intervals. This on-demand infrastructure can be deployed in minutes to so that jobs run only when needed and cease when the update is complete. As a result, companies only pay for the actual time that the clusters are active.

Hybrid data integrations to avoid disruption

Once your company or organization decides to move your data to the cloud, one big consideration will be how to maintain your current data warehouse until the integration process is complete. By using AWS Redshift in tandem with your on-premise data warehouse, it’s possible to create a hybrid data storage solution that reduces costs and improves agility, without disrupting your operations. Your data integration tool should include connectors that allow you to migrate your data with AWS Redshift seamlessly, predictably, and securely.

Most cloud-based solutions include hybrid integration capacity, and a comprehensive data integration tool should include a variety of connectors to bring your data migration jobs to completion, no matter where your data is stored.

Examples of data integration at work

Up to this point we’ve looked at the process of integrating with AWS, along with some of the reasons why companies choose to migrate their data. We’ve also considered the process of data integration and how the right data integration tools can help provide a seamless transition and improved efficiency. But what does the data integration process look like from the perspective of a real company with real challenges? Here are two examples:

Integrating with AWS to reduce costs by 75%

Healthcare company Accolade had access to mountains of data and wanted to use it to recommend personalized services to their customers and streamline their operations. Much of the data was siloed in legacy systems, but Accolade knew that to get the most out of their data, it would need to be transformed, migrated, and integrated. They needed a comprehensive solution that could map, decrypt, and profile data before migrating it to a data lake for integration with AWS.

By connecting all of their data with Talend Big Data Integration, Accolade was able to use AWS Redshift, S3, and EMR to improve efficiencies and provide better care for their patients. By enriching and applying cloud analytics to their data, Accolade was able to lower healthcare costs for their patients by 5-8% annually and drive a 75% cost-reduction in their patient onboarding process.

Expanding access to education through data integration

The University of Pennsylvania offers more students access to high-quality education through its no-loan financial aid policy, which allows students to avoid amassing large amounts of education debt while they complete their studies. To make the policy feasible, the University relies on an extensive network of 300,000 active donors.

The university faced two challenges. First, they needed to integrate data from multiple CMR systems into a single location. Second, they wanted to ensure that they were taking advantage of every cost-saving measure possible, including scalability and elastic deployment. With Talend Cloud, UP was able to integrate data from multiple sources and mine it for insights that translate into better relationships with their benefactors. The result: a 7% increase in the number of gifts and an 18% boost in revenue.

Getting integrated with AWS

Planning and executing an AWS integration strategy may seem like a daunting task, but it doesn’t have to be. With the right integration tools and information, anyone can get their integration project underway quickly and reliably.

Talend Cloud Integration Platform helps you manage on-premises, cloud, and hybrid integrations with AWS. Powerful graphical tools, integration templates, and over 900 components are at your command to make sure your integration is a success.

Start your free trial and get your hands on everything you need to get to AWS today.