DBmaestro’s 2019 Database DevOps Survey confirmed that Database Configuration Drifts are one of the biggest problems developers, DBAs, IT professionals, and other stakeholders are facing today. More than 70% of the respondents admitted that Configuration Drifts created issues and errors while making changes to their databases, a statistic that speaks for itself.

What is a Database Configuration Drift?

Database Configuration Drifts are a huge roadblock when it comes to establishing smooth DevOps pipelines. Any kind of inconsistency can elevate the risk of operational failures and create deployment issues. Furthermore, it is also hard to pinpoint the cause of the problem due to multiple and incremental versions, something that creates even more delays and bottlenecks.

Also known as Version Drifts, these issues occur when the version in production is not exactly what was intended. This happens when the database is patched or updated without deploying it properly, instantly creating a blind spot and inconsistency between the source control and the database. This often has a snowball effect with multiplying consequences as newer versions add up.

A Production vs Source Control Drift

The ideal scenario should involve pre-specified pre-deployment checks to make sure that the version in production is indeed the right one. Once an issue is detected, it should be addressed instantly with a quick roll back or version update. Understanding how these changes may have entered production is one of the biggest challenges organizations are facing today.

Here are some examples of why Database Configuration Drifts may occur:

  • Critical Package Updates – Modern apps that are business critical are typically built upon a smaller package frameworks. These packages may be developed internally or by a third-party vendor. If a security or performance issue is found in one of these packages, a developer may opt to update it in real-time without following the official protocols.
  • New Server Installation – Organizations are constantly making infrastructure changes while scaling up. Let’s assume a new production DB server has been installed and all EAP servers need to be updated so that the URLs refer to the new one. But if you have an old server that hasn’t been updated, you will start getting programmatic errors.
  • Scheduled Password Changes – Password changes are done in a scheduled manner in most organizations looking to secure their database today. However, this can create serious issues if ad-hoc error-prone automated scripts are used to do so, creating inaccurate data source configurations. Sometimes, not all servers are updated.

With Continuous Integration (CI) and Continuous Delivery (CD) pipelines promoting incremental deployments on a daily basis, lack of source control governance can quickly lead to multiple Database Configuration Drifts that can be very hard to remediate due to the dynamic nature of development teams and the large number of stakeholders making changes to the code.

Problems Caused by Configuration Drifts

Making deployments the right way involves taking a series of steps and documenting all changes. But as explained in the previous section, the dynamic nature of today’s DevOps pipelines is leading developers, IT professionals, QA Testers, and DBAs to take shortcuts and create undesired inconsistencies. Add the lack of documentation into the blend and you have a recipe for failure.

  • Security Breaches – Unwanted, unmonitored, and undocumented database changes can open up your internal infrastructure to malicious external access. These changes can be quite harmful, like opening a port for outside access or by installing malware on a vulnerable machine, which is not difficult due to the BYOD working patterns in place today.
  • Poor User Experience – Poor Database Drift Configuration detection and remediation can be the difference between slight inconvenience and uncorrectable brand damage. These database issues can create performance issues and disabling of features, which has a direct impact on customer satisfaction levels and eventually also your bottom line.
  • Denial of Service – One Database Configuration Drift can have a huge chain reaction leading to unexpected downtime in many cases. Besides the obvious operational and business implications, fixing these issues can take hours or even days. When changes are not documented and recorded, remediation becomes a long and cumbersome process.

To sum things up, there is a glaring lack of visibility into the changes that are being made and their impact on the database. These changes affect each other and affect the system in unexpected ways that cannot be forecasted or planned for. Without proper documentation and version management, these changes often go undetected until the worst happens – system failure.

Preventing Configuration Drifts

Traditional methodologies and processes are becoming more and more ineffective in preventing Database Configuration Drifts. This is mainly because of the manual steps that simply cannot be taken properly – either due to lack of manpower or due to shortage of time. The modern organization needs to adopt a proactive approach and adopt modern methodologies to mitigate drifts.

  • Dedicated Policy Enforcement – Hotfixes and patching are big parts of all DevOps pipelines today. But not having a predefined policy to make these can lead to multiple drifts that can create huge problems down the line. You should ideally create a clear and visible path for app and DB code, with a clear testing stage baking into it to catch issues, if they arise.
  • Promote Dry Runs – Your developers and other related stakeholders should be encouraged to perform dry runs. There are basically safety nets that simulate releases before they hit production and go live. Not only does this minimize the risks involved but it also helps optimize performance and security levels, while allowing proper documentation.
  • Roles and Permissions – Today’s development setups have become extremely dynamic, with professionals being hired at a rapid pace and working from multiple remote machines. To put it bluntly, the modern DBA cannot possibly track and monitor access to databases like before. There is a need for some kind of automated governance and management tool.

Taking these steps will help you eliminate Database Configuration Drifts from your ecosystem, but you will also need automation to connect the dots.

Related: Zero Downtime Database Deployment

Database Automation to Eliminate Drifts

Preventing Database Configuration Drifts is crucial because “time is money”. Developers and database professionals should be productive, while focusing on innovation and quality. But if they spend time on fixing drifts and inconsistencies, your product will suffer from multiple performance, security, and quality issues. These problems will often multiply when you scale up.

By using a comprehensive database automation solution, you will be able to optimize your migration-driven database delivery. This methodology tends to be faster than state-driven database delivery, which doesn’t really match the DevOps philosophy, but still has some vulnerabilities in the deployment stage. For example, your script can override changes made by others.

However, these issues can be nullified by using a solution that can map and understand the structure and configuration of your databases. By doing so, configuration baselines are tagged automatically, before and after the introduced changes. It gets better. The target environment’s configurations are then validated prior to the release to make sure that no drifts occur.

Needless to say, all actions are documented automatically, making the DBAs life easier. No more manual methods and procedures to track what is going on. This is also a huge advantage when it comes to creating audit trails. With GDPR, HIPAA, and CCPA regulations becoming increasingly tighter, you need to make sure that compliance functionality is baked into your automated ops.

Summary

Automation is no longer a choice for the modern organizations looking to establish smooth Database DevOps pipelines. The traditional manual methods and processes look good on paper, but simply cannot be implemented due to the dynamic development characteristics such as incremental releases and testing. Things get even more complicated while scaling up fast.

Automation is the only thing standing between you and your next database configuration drift, which even the most effective best practices can’t prevent.

Implementing a comprehensive database automation solution can help you eliminate release breaks and downtime created by configuration and version drifts. This is the most effective and cost-saving way to make sure your applications are released fast, with no performance and stability flaws that may harm customer satisfaction and impact your revenue streams.