IaC-CICD-ARMTemplate-AzureDataBricks-Sample - Showcase Project

Infrastructure as Code Showcase Project using ARM Templates and CICD for Azure Databricks

CI/CD Sample using ARM Template Deploying an Azure DataBricks Workspace with VNET Injection

Azure DevOps Project

https://dev.azure.com/mertsenel/Infrastructure-as-Code-Samples

Credits

The base ARM template referenced from Azure quickstart templates:

https://github.com/Azure/azure-quickstart-templates/tree/master/101-databricks-workspace-with-vnet-injection

Builds

https://dev.azure.com/mertsenel/Infrastructure-as-Code-Samples/_build?definitionId=5&_a=summary&view=buildsHistory

Releases

https://dev.azure.com/mertsenel/Infrastructure-as-Code-Samples/_release?definitionId=3&view=mine&_a=releases

Details of the Project, Improvements and Additions on top of the base ARM Template:

Project:

• ARM Template has been modified to support Environment paramater, this would allow multiple SDLC enviroments can be deployed via same template, with minimum input as namings are generated within variables block via the input of the paramater.
• The environment abbrevation ex. dev, qa, test, staging, prod is automatically appended to naming conventions of resources and resource groups.
• This structure allows dynamic paramaterization of the CD pipelines.

(The project was initially just deployed a standalone DataBricks Workspace, later on I’ve converted the project to deploy Azure Databricks workspace with VNET injection. I will leave development branches in case you want to see my work, history is also available through Pull Requests)

Azure DevOps Pipelines:

Continuous Integration

• Yaml based pipeline built for validating the ARM template and archiving template and parameter files to be consumed by Release Pipelines.
• A Pester test script is also added, for a second verification with the Azure Powershell AZ Module’s Test-AzResourceGroupDeployment and evauluation ouput of this cmdlet.
• TODO: Although Pester test script is successfully passing its test while running locally, it is unable to get a succesfull pass result while running on hosted agents of the Azure DevOps. To be Fixed.

Local Pester Test Run Screenshot:

• Pester Tests are now fixed for CI. Results are published in NUnit format and published to the Build CI Pipeline.

Pester Tests Run Screenshot from Build CI:

Sample Build Pipeline Screenshot:

Continuous Deployment

• Enabled Continous Deployment for both Pull Request and Push(to master and release branches) to Dev environment.
• This environment and Release stage can later be used to run integration tests before promoting deployment to higher stages.
• Created a Release Pipeline to have Dev and Test and UAT Stages, with stage specific “env” variables to pass to ARM Deployment task to override the “env” parameter of the ARM template.
• Have seperate ARM templates for prerequisites resources and main DataBricks resource. Release Pipeline variables are used to pipe output (VNET Resource ID) to the main template as a paramater.

Release Screenshots

Sample Release Pipeline

Sample Release Pipeline Stage

Code Management / GitHub

• Enabled master branch protection in GitHub repository with status checks requirement for safe contribution.

Deployed Resource Screenshots in Azure After Release

Below screenshots are to demonstrate naming convention and how easy it is to scale projects for different stages with dynamic paramaters.

Resource Groups

All Resource Groups after Successful Release

CI (Manual Run from Local)

DEV

TEST

UAT

Other Screenshots

Sample VNET Delegation

Sample NSG After Databricks Delegation