The recently held “End-to-end CI/CD with Snowflake and DBmaestro”, hosted by Yaniv Yehuda, CTO of DBmaestro, shed some light on this very important topic. We invite you to access the on-demand recording while also reading on to catch the key takeaways. DBmaestro and Snowflake, when used together, will help you achieve Database DevOps for optimal results.
Database DevOps: Moving Faster
Discovering requirements and developing solutions on the go requires not just a cross-functional team but also a great deal of adaptability. Everyone wants to move faster than they do. Agility in such situations is essentially what you need. This is making the world of software development an obstacle course that you have to run at breakneck speed with limited vision.
DevOps is not just about setting up development processes or creating a workflow that helps you combat operational issues. It is a set of principles and practices that focus on the people and their approach to solving problems. It enables everyone in your team to collaborate and come up with relevant solutions in unique or unprecedented situations.
How does this translate to real-life? For your DevOps team, a key indicator of how well they work is their ability to deploy at will. In a 2019 report by Accelerate State of DevOps, elite teams deploy several times a day, high performing teams anytime between once a week or once a day, and low performing teams deploy anywhere between once a month or once every six months.
The Database is Being Left Behind
Unfortunately, database professionals are still not always in the loop. There are three main reasons this is happening today.
- Data is complex: Apart from databases being massive, complex, ever-growing entities, they are also hard to manage. DBAs and database teams, while playing a critical role in the CI/CD pipeline, are usually cut off from architectural decisions, which isn’t great considering they are the ones who control, track, and audit each change.
- Database professionals have a very specific job: Database professionals prioritize the integrity and stability of the data over the developers’ need to change things. However, this is in direct conflict with what businesses want- moving faster. This trend has redefined the role of DBA’s from the security guys to spending 37% of their time on making DB changes which less than two years ago took just 22% of their time.
- Database professionals have a very specific job: It does not help that DBA’s are often looked at as the rigid and strict professionals who take a little too much time to deploy changes. Being the single point through which changes flow adds to the work pressure that DBA’s face with frequent deployments being requested by the development and the operations team.
In a 2020 survey, it was found that 18% of the companies faced a significant problem with their database in the last week and 50% faced similar problems in the last 3 months. Interestingly, 80% of outages impacting mission-critical services are due to human error, where 50% of these outages happen due to change configuration, release integration, and hand-off issues.
Did You Know?
Database downtime can cost your organization up to $7,900 per minute, not including brand damage and customer churn.
Move Faster Without Breaking the Database
Firstly, you need to make sure that all DevOps best practices are also applied to the database, which is often not the case. This is because you are creating a repeatable process, with integrated security and compliance policies baked into it. Furthermore, you should ideally be handling roles and permissions in an automated manner and focus on what matters most – dry-runs and testing.
When it comes to Database DevOps, the key lies in testing the changes before applying them in the production environment. Having a simulation of the data can help you to remove bottlenecks and stop you from blocking a table for too long. Once your code is tested and ready to be deployed, you have to decide whether to use the state or migration-based deployment.
State-based deployment offers immutability and reliability in testing, but is unable to repeat the change process in another environment. Migration testing offers that repeatability but requires the user to build the code manually. While organizations had to choose between the two before, database automation has created the option of combining the two methodologies for best results.
Read More: State vs Migration Database Deployments
Database DevOps Best Practices
A market that will be worth over $1700 million by 2023, database release automation allows you to solve many of the issues that crop up due to human error. Automating database releases will allow your business to accelerate release cycles and improve performance by bringing focus to innovations while being in harmony with the dev teams, IT teams, and other stakeholders.
Let’s run through the three biggest advantages of embracing Database DevOps.
Security and Policy
Regardless of your specific position, your organization needs to know the answers to the following questions regarding your database at all times.
- Who is provided with access to the database/s?
- Who can transfer versions over from one environment to another?
- What are the policies or procedures for VPN network users?
You need to automate tiresome tasks such as setting roles and permissions for each individual user, specifying the precise access and type of access that is granted to each team member. Enforcing separation of duties and specifying exactly which projects and environments can be changed and by whom are questions best answered right at the beginning of your project.
Furthermore, you need to be able to customize your policies for issues that may crop up later in the process. Are all the channels of communication between team members secure? These are inevitable questions that will need quick thinking and coming up with solutions that also support the addition of custom, project-specific policies, or global company-wide policies.
Compliance and Traceability
Data privacy has taken center stage in recent years. You cannot possibly achieve sustainable business growth if you are not GDPR (General Data Protection Regulation) compliant in Europe or HIPAA (Health Insurance Portability and Accountability Act) compliant in the USA. There is also Sarbanes-Oxley (SOX) Compliance that has to be taken seriously today.
Leaving compliance procedures to a few team members creates massive bottlenecks. The lack of self-service only creates a host of productivity issues and brings the CI/CD process to a halt whenever the database is involved. Adding to such woes – permissions are managed manually with no centralized control and low visibility into who has what permissions and why.
When permissions are configured via roles, you don’t need to manage the role permission at the user level. Also, all audit trails are created automatically.
Automate and Optimize
Automating database releases and deploying the same way to all environments will save you a lot of time and effort. But it’s not just that. You will start enjoying the benefits of traceability and logging, while embedding risk management procedures into your CI/CD pipeline. You will have the time to perform dry-runs and locate issues before the bad code reaches production and causes issues.
You will also be able to eliminate database version configuration drifts. These inconsistencies are one of the biggest problems while enforcing Database DevOps today. As documented in the DBmaestro 2019 Database DevOps Survey, over 70% of the respondents admitted to encountering this technical issue that creates release bottlenecks, performance problems, and unwanted downtime.
Read More: Demystifying Database Configuration Drifts
End-to-End CI/CD with Snowflake and DBmaestro
DBmaestro is partnering with Snowflake to create smooth and bulletproof CI/CD pipelines. Snowflake, an American cloud-based data-warehousing company, offers out-of-the-box features like storage segmentation, seamless scaling up capabilities, and third party integration. This is where DBmaestro enters the picture and connects the dots to provide a solid Database DevOps offering.
The DBmaestro platform serves as an end-to-end database solution that sits on top of Snowflake (and other vendors like Oracle, PostgreSQl, and more). With DBmaestro you can automate and manage your DB release pipelines, create pre-deployment dry-runs, kick out configuration drifts, and manage roles and permissions. Everything is accessible via a centralized dashboard.
Yaniv wrapped up the webinar with a live demo of DBmaestro working on top of Snowflake to create a user-friendly environment for the DBA. He showed how user and permission management can be automated, while also performing other Database DevOps relation actions via the solution’s centralized dashboard. Click on the link below to access the on-demand webinar.
