Database Modernization: Key Lessons Learned

We’ve learned many lessons over the years on how to execute successful database modernizations. These projects are rarely ever straightforward. The world is full of talented database engineers and developers, yet many modernizations fail. Why? Because experience matters tremendously in this area.

Database teams must have a holistic understanding of what it means to modernize a database. This includes knowing how to effectively serve the database, the development team, and the overall DevOps function. Without this comprehensive approach, database modernizations fail or take much longer (years instead of months).

As an AWS Premier Tier Services partner, ClearScale has seen everything when it comes to modernizing databases on the AWS cloud. So, we’re here to share six key lessons that we’ve learned along the way and to dispel any common database modernization myths.

Lesson 1: Determine Appropriate Test Cases in Advance

We initially expected our clients to take the lead on the testing side of things. However, we’ve since learned that we can’t rely on customers for appropriate test cases. Many clients aren’t ready or haven’t thought enough about their database modernization testing strategy.

That’s why we recommend spending more time discussing testing during the early stages:

• How will you test the target application?
• What are the most critical test cases?
• Who will do the testing?
• Do those people have the right skills and capabilities?
• Is automated testing included in the plan?

Answering these questions in advance is crucial for ensuring modernization success and keeping things on schedule.

Lesson 2: Establish the Development Environment

The development environment is another piece that often gets overlooked in database modernizations. We’ve been in situations where we’ve left the development environment out of the project’s scope, thinking the client had it under control. After diving in, we were then surprised to find nothing was ready on the development environment front. As a result, we’ve had to create change orders to do the work ourselves. We now make sure to validate the status of the development environment early on before getting started.

There are two important factors here – the environment itself being suitable for development (IaC, CI/CD, etc.), as well as data that are used during tests. This can be a test environment, rather than DEV, but it’s still very important for development teams.

Lesson 3: Don’t Assume the Customer Understands the App

We’ve also learned in our database modernization experience that some companies  don’t know everything about how their applications work. IT teams change, and applications grow increasingly complex over time. You can’t assume your clients know the inner workings of their target applications. This, of course, makes it difficult to estimate timelines and costs.

So, it’s important to validate that clients have a deep understanding of their applications and databases. If they don’t, your timeline estimates should be at least 2-3x 5-10x longer. Our general philosophy is to double-check everything based on our own application assessments.

Lesson 4: Define the Cutover Time in Advance

We’ve assumed in the past that cutovers would be quick and seamless, similar to how they are with standard migrations. We generally schedule cutovers during a single weekend for those types of projects. The problem is that database modernizations never follow this pattern.

The main reason is that your customers have data for all of their own customers. Therefore, they usually want to execute cutovers in phases, like 10 weekends in a row versus getting it all done in one weekend. Practically speaking, this requires more PM hours and a longer project for your team to execute. If the cutover strategy is not discussed in advance, you may end up writing many change orders and spending far too long on one engagement.

It’s important to stress that a “big bang” migration – in which applications and data are all migrated in one operation – significantly increases risk. Phased cutovers are the best practice.

Lesson 5: Adopt a Flexible Staffing Model

The resources you need for a database modernization project change as you progress. You’ll often work with the customers for several weeks, or even months, to first scope and prep the project.

This means you don’t have to onboard database developers right away, which is what we used to do. You can wait until you and the client absolutely need those resources, allowing you to keep those experts busy on other projects.

The same thing happens at the end of projects. Database engineers may continue working, but the application development team is not needed anymore.

Lesson 6: Performance Testing is Essential

Years ago, when we were new to database modernizations, we assumed that if we migrated from something like Oracle to PostgreSQL, we would only need to modernize the underlying codebase. The truth, however, is that adopting a new database solution doesn’t automatically lead to better performance.

Some features that work quickly in Oracle won’t in PostgreSQL. That’s why performance testing is paramount, especially given that database migrations and modernizations are commonly the first time in years that organizations test systems in their entirety. Even more, these tests may lead to the need to rearchitect certain business processes because they will not run fast enough on the target database.

The Ideal Database Modernization Customer

The benefit of knowing these lessons is that you can avoid issues and challenges that would otherwise delay database modernization projects. At ClearScale, we go into every project with these lessons in mind and work through potential obstacles with our clients, leading to better results across the board.

Additionally, we’ve learned how to identify the ideal database migration and modernization customer. For Oracle to AWS migrations, the perfect client is one who has already figured out their total cost of ownership and has clear reasons for why they want to migrate to AWS. Many companies don’t do this, which can be a showstopper because we aren’t able to calculate the difference in licensing costs on their behalf. And licensing contributes to about 90% of TCO. But, if a customer comes to us with a defined business case, we can guide them through how to do everything else.

On top of having a well-defined business case, great clients also already use modern development practices including CI/CD, blue-green deployments, and automated testing. They are also willing to pause feature development for the duration of the migration. This makes things easier.

Real World Examples

Let’s talk about some real-world examples that highlight the ways in which database modernizations can go sideways.

Health and Wellness Company

We worked with one company in the health and wellness industry that was expecting a fixed-timeline, fixed-cost engagement. For many reasons, this didn’t happen:

• The customer’s CI/CD infrastructure required additional attention to scale and support multiple versions of application deployments.
• There was concurrent development between the customer and ClearScale engineers, which meant we spent considerable time merging code between the two branches. SQL code also was not version-controlled, making it harder to merge code changes.
• We originally scoped a “big bang” migration cutover. We had to revert to a phased approach (a best practice for larger clients), which doubled the anticipated timeline.
• The client didn’t have dependable automated testing for the entire system. This increased the risk that something wrong would happen in production after the cutover.
• The client’s system was highly configurable, which made it hard to account for certain use cases in non-production environments.
• The project schedule was too compressed.
• Database performance optimization requires time to solve performance issues and the need to rearchitect certain business processes.

Digital Advertising Company

In another engagement, we were working with a client in the digital advertising space that experienced the following issues:

• The company had never set up a separate sandbox environment, which complicated and delayed the testing efforts. For instance, some of the tests that the team ran in the sandbox environment affected external integrations.
• The client decided to migrate analytical workloads to a data lake and OLTP workloads to another database engine at the same time. This introduced too many dependencies that had to be handled perfectly at the right time (i.e., another “big bang” release). Ideally, these migrations would have been performed sequentially to minimize risk.
• The client lacked a dedicated testing environment, including for the database.

Healthcare Company

Finally, we worked with a company in the healthcare space that ran into the following snags:

• The client had lots of tribal, in-grained knowledge that required significant ad-hoc discovery as our developers were making changes, deploying, and testing code.
• Key individuals were not available full-time at the beginning of the project to onboard our database modernization team.
• Migration efforts could not be localized to the core application and database. A lot of supporting infrastructure and tooling also had to be updated, including a custom installer for the on-premises app that no one had touched in 4-5 years.
• We had to dedicate major effort to testing migrated queries, which included using static analysis of dynamic queries using the JSQL parser on the application side. This ensured that queries were being constructed correctly.

Our hope is that these real-world experiences reveal the many ways that database modernizations fall behind or fail altogether. Use these lessons to keep your database modernization projects on track.

Want to learn more about ClearScale’s AWS database services?

Get in touch today to speak with a cloud expert and discuss how we can help:

Call us at 1-800-591-0442
Send us an email at sales@clearscale.com
Fill out a Contact Form
Read our Customer Case Studies