Reducing Data Warehouse Costs and Increasing Scalability

Today, many organizations are migrating on-premises data infrastructure to the cloud to reduce data warehouse costs and increase scalability. One institution began its cloud migration when its custom-built Oracle database couldn’t support its advanced research initiatives. Now, the organization has a cloud-based data infrastructure that can scale and adapt as needed over the long term.

The Challenge

The organization originally hired a third-party software development firm to design and host its on-premises data warehouse using Oracle RDBMS to support research needs. The solution worked well when its datasets were smaller and its data requirements were less complex. However, the on-premises solution began to fall short in several ways.

The organization was unable to support multiple types of workloads, including regular and analytical queries. Consequently, the IT team had to keep high-compute configurations running at all times, despite not needing all of that processing power for most tasks.

Also, the on-premises database wasn’t scalable, elastic, or stable enough to support all of the organization’s ambitious research goals. Furthermore, the on-premises architecture was costly to maintain, forcing IT to spend more than needed for data warehousing.

After evaluating different options, the organization decided to move some of its resources to the cloud. The institution wanted to bring in a cloud expert that could design and implement a new cloud-native data warehouse. They found the right partner in ClearScale.

The Solution

ClearScale developed several proofs-of-concept (POCs) to demonstrate the value of migrating the client’s on-premises data warehouse to AWS.

ClearScale created one POC to replace on-premises Oracle instances with Amazon Relational Database Service (RDS) instances without any ongoing replication. To do this, ClearScale proposed a DB migration and dump file process.

An engineer on the client side prepared an Oracle DB dump and uploaded the file to Amazon S3, a highly durable object storage service. A ClearScale engineer then created a new RDS Oracle instance on the cloud side.

Amazon RDS allows users to create instances through the S3_INTEGRATION approach, which enables users to point to a source dump file and restore an entire database in the cloud. The RDS instance itself was put in a private subnet to only permit direct access through a VPN connection. Once the database was live, ClearScale moved to the next phase.

The second proof of concept used several AWS services, including:

Amazon Redshift
AWS Glue
Amazon Athena
Amazon QuickSight

ClearScale used Amazon Redshift, a cost-effective, cloud-native data warehouse for the data migration. For the data transformation pipeline, the team leveraged the fully managed ETL service, AWS Glue, which comes with crawlers that scan data stores and propose data structures that work well within the AWS Glue Catalog.

To support the customer’s data query needs, ClearScale implemented Amazon Athena, a serverless, interactive query service that can evaluate data stored in Amazon S3. Finally, ClearScale set up Amazon QuickSight. With Quicksight, the client’s researchers can create interactive dashboards to publish analytical insights, including those generated by machine learning algorithms.

The Benefits

With help from ClearScale, the customer’s data warehouse is more flexible, elastic, and durable, setting the stage for future cloud optimization projects. ClearScale migrated much of the on-premises data warehouse to the cloud, allowing the customer to take advantage of AWS’s flexible pricing, automated scaling, and increased reliability.

ClearScale’s design approach allowed the client to maintain the storage and compute power needed for its current workloads at a much lower cost. Additionally, the cloud-based assets can now scale horizontally (via read-replicas) and vertically (via instance size upgrades) without requiring IT intervention.

Looking ahead, the organization has the data infrastructure it needs to scale with its ambitious research initiatives while benefiting from pay-as-you-go pricing.

To learn more about cloud data migration, download the eBook.

Get in touch today to speak with a cloud migration expert and discuss how we can help:

Call us at 1-800-591-0442
Send us an email at sales@clearscale.com
Fill out a Contact Form
Read our Customer Case Studies