Infrastructure as Code CI/CD ARM Template Azure DataBricks Sample
IaC-CICD-ARMTemplate-AzureDataBricks-Sample - Showcase Project
Infrastructure as Code Showcase Project using ARM Templates and CICD for Azure Databricks
CI/CD Sample using ARM Template Deploying an Azure DataBricks Workspace with VNET Injection
Azure DevOps Project
The base ARM template referenced from Azure quickstart templates:
Details of the Project, Improvements and Additions on top of the base ARM Template:
- ARM Template has been modified to support Environment paramater, this would allow multiple SDLC enviroments can be deployed via same template, with minimum input as namings are generated within variables block via the input of the paramater.
- The environment abbrevation ex. dev, qa, test, staging, prod is automatically appended to naming conventions of resources and resource groups.
- This structure allows dynamic paramaterization of the CD pipelines.
(The project was initially just deployed a standalone DataBricks Workspace, later on I’ve converted the project to deploy Azure Databricks workspace with VNET injection. I will leave development branches in case you want to see my work, history is also available through Pull Requests)
Azure DevOps Pipelines:
- Yaml based pipeline built for validating the ARM template and archiving template and parameter files to be consumed by Release Pipelines.
- A Pester test script is also added, for a second verification with the Azure Powershell AZ Module’s Test-AzResourceGroupDeployment and evauluation ouput of this cmdlet.
TODO: Although Pester test script is successfully passing its test while running locally, it is unable to get a succesfull pass result while running on hosted agents of the Azure DevOps. To be Fixed.
Local Pester Test Run Screenshot:
- Pester Tests are now fixed for CI. Results are published in NUnit format and published to the Build CI Pipeline.
Pester Tests Run Screenshot from Build CI:
Sample Build Pipeline Screenshot:
- Enabled Continous Deployment for both Pull Request and Push(to master and release branches) to Dev environment.
- This environment and Release stage can later be used to run integration tests before promoting deployment to higher stages.
- Created a Release Pipeline to have Dev and Test and UAT Stages, with stage specific “env” variables to pass to ARM Deployment task to override the “env” parameter of the ARM template.
- Have seperate ARM templates for prerequisites resources and main DataBricks resource. Release Pipeline variables are used to pipe output (VNET Resource ID) to the main template as a paramater.
Sample Release Pipeline
Sample Release Pipeline Stage
Code Management / GitHub
- Enabled master branch protection in GitHub repository with status checks requirement for safe contribution.
Deployed Resource Screenshots in Azure After Release
Below screenshots are to demonstrate naming convention and how easy it is to scale projects for different stages with dynamic paramaters.