Leveraging Rsync and Rundeck to Replicate AWS EFS

Since its inception, Amazon Web Services has created numerous Cloud-based services to give its clients the ability to create phenomenal architectures and applications with minimal investment and high scalability and redundancy. However, some of those services provide the backbone of basic functionality just enough to make it useful but not necessarily enough to meet each and every need of clients that rely on them. In times like this, it falls on customers of AWS to find alternative and creative approaches to overcome some of the perceived gaps in functionality.

This was true for a recent client that needed to find a way to transfer files to a second region. Amazon Web Services created a heavily used service called Elastic File System (EFS) designed to serve as a centralized repository for any files that needed to be shared across multiple AWS EC2 instances. This allows EFS to provide common file sets to any number of workflows and applications running across many different EFS implementations. Because of the need for it to remain flexible to meet any number of client needs, AWS designed the service to mount to on-premise data center servers or be hosted in the cloud so that content could be backed up there as needed.

The Challenge – Data Replication for Disaster Recovery

A recent client of ClearScale’s needed to take the concept of EFS beyond the out-of-the-box functionality that Amazon provided. Because the client provided a heavily used micro-donation processing application to Fortune 1000 customers for use in managing donations, grants, and volunteer activities, the need to replicate the data stored in AWS Elastic File System was critical to the ongoing success and growth of their Big Data platform.

Although it allowed for one-time backups of the content to another region using EFS File Sync, EFS did not provide the ability to actively replicate the data to a second region on a consistent basis due to its design limitations. The client needed to find a way to accomplish this for disaster recovery needs and so engaged with ClearScale to see if a solution could be found.

The Solution – Rsync

ClearScale developers approached this problem by getting back to the basics. At the heart of the issue was the need to transfer files from one region to another in order to address the need for disaster recovery. The existing EFS File Sync merely transferred files at a given point in time and copied the content and metadata. There was no built-in functionality to identify deltas in the information that had been added, deleted, or modified and then have that information transferred over, which was a basic fundamental requirement for any disaster recovery need.

To accomplish this feat, ClearScale looked to Rsync, a tool that synchronizes files and directories between two different systems. Once invoked, Rsync will communicate between the host and the secondary environment to determine what files diverge from each other. It can then, in conjunction with EFS File Synch, transfer the necessary files over to the secondary location.

EFS Replication Diagram

However, this Rsync service does not simply activate whenever changes occur in the host system. For that, ClearScale turned to Rundeck, an IT Operations management platform. Built as a generic scheduling and orchestration platform, ClearScale chose to use this tool to provide a method to kick-off the Rsync process. This allowed the client’s administrators to then schedule on a cadence how often Rsync would identify differences between the primary instance and the secondary instance and then transfer the files over as needed. Further, Rundeck allowed for monitoring the disaster recovery instance and in turn, send notifications out should any issues arise. This Rsync/Rundeck implementation thus allowed for a cohesive solution by moving files over on a regular basis to support a true disaster recovery model.

The Benefits

This methodical approach to file transfer allowed ClearScale to enact a solution to the customer that would allow them to replicate and maintain a secondary application environment in an entirely different geographical region, thus allowing them a measure of disaster recovery. In the absence of functionality that would make AWS EFS File Sync a truly powerful tool, ClearScale demonstrated that using off-the-shelf proven technologies could accomplish what EFS File Sync could not. This allowed the client to further build out and scale their micro-donation platform and, at the same time, provide their Fortune 1000 customers peace of mind that, should anything happen to the primary instance, a proper disaster recovery region would be enabled with minimal impact to their operations.

Since 2011, ClearScale has found unique approaches to any number of sizable challenges. Sometimes, it requires brand new architectures and home-grown services or third-party tools to interact in unique ways. Other times, it means getting back to the basics and using proven technology to solve challenging scenarios. ClearScale has shown time and again that the competencies that AWS has bestowed demonstrate our ability to not only understand the complexities of the AWS service landscape but to augment these services with creative approaches designed to meet any client-critical need.

Get in touch today to speak with a cloud big data expert and discuss how we can help:

Call us at 1-800-591-0442
Send us an email at sales@clearscale.com
Fill out a Contact Form
Read our Customer Case Studies