Platform Engineering

Ross McDermott

Establishing Guardrails with Azure DevOps Pipelines

Posted by Ross McDermott on 03 March 2021

Microsoft, Delivery, Microsoft Azure, Azure Pipelines, Azure DevOps, cloud computing

With the constant evolution of governance, compliance and security, it can be a daunting prospect to balance best practices while promoting lean and efficient delivery approaches.

In this post, we will walk through and demonstrate how using best practices pipeline structures, standardisation, and cross-team collaboration allows us to establish guardrails to support our build and release processes. The codification of these guardrails provides comfort to stakeholders that their obligations are being considered and enforced, and delivery teams can be confident that they are building and deploying in a safe and compliant way.

What is a guardrail? Like the safety barriers that keep us safe when driving, guardrails in a pipeline aim to make the processes we use to build and release safe by ensuring required tasks (technical, governance, compliance or security related) are implemented in an agreed way. Embracing a guardrail approach means teams can build and release with confidence knowing they are aligned to, and supported by the organisational requirements for safe delivery.

Note: There may be scenarios where teams need or want to approach a problem in a different way to that defined in a guardrail. The intent of a guardrail is not to dictate the only approach, but to define an agreed approach. When proposing an alternative approach teams will likely need to seek approval from the respective stakeholder(s); this may be fed back into the agreed guardrail approach, or teams may be asked to align with the existing approach.

Setting the scene

To help give some context to the problem and approach, let’s explore a scenario that we will apply in the proceeding design and discussion.

As part of a new initiative, a series of containerised .NET Core services will be developed and deployed to Azure App Services. Containerisation has been chosen as the packaging approach to provide flexibility for future hosting decisions and promote standardised delivery. Configuration for services will be managed though Azure App Configuration to ensure configuration is externalised from the application logic.

The following is a logical view of the target state Azure resources:

Logical Architecture

 

Within the context of the scenario, let’s assume we have had workshops with security, compliance and operational teams, and the following have been identified as core objectives:

  1. The build and deployment steps for each service must be consistent and centrally defined to allow tasks to be added or removed over time.
  2. Unit tests must run and pass as part of the build process to drive quality.
  3. Container scanning must be incorporated, with any detected medium or higher severity results blocking the build / release process.
  4. To support audit and change management requirements, only specific permitted pipelines should be granted permission to deploy into environments.

If we can establish supporting guardrails for the above objectives, we can provide comfort to our stakeholders, and teams can focus on delivering business value.

Azure DevOps Features

We will be leveraging several Azure DevOps features as we work through the above scenario, each provides us a part of the overall puzzle to establish our guardrails.

Repository Resource Allows us to reference multiple repositories within our pipeline to reference shared artefacts.
Extending from template Provides us with the ability to centrally define and manage a set of pipeline definitions and extend from those base templates for each service we develop. This will enable high reuse and low coupling between services and the centralised pipelines.
Required Template

Enables us to apply a check that ensures only pipelines that have extended from allow-listed pipelines are permitted to be deployed.

This provides us a level of security that only our agreed base pipeline templates can be used when deploying, ensuring our guardrails are not being bypassed.

Approvals Ensures that specific users or groups have explicitly approved the release into an environment.

Pipeline Design

Now we make an assumption we are using a multi-repo design for our initiative - with a repository per service and a ‘service-base’ repository to hold shared assets for the pipelines. The same pipeline approaches can be used in both multi-repo and mono-repo designs.

Repository Structure

 

A copy of all pipeline templates and demo service can be found here.

Expanding from the repository structure, the below describes the key templates in our design that will enable the implementation of the guardrails.

Pipeline Repository Structure

 

1. Orchestration

Orchestration between the build and release stages within the pipeline is managed within the azure-pipeline.ci-cd.yaml template. The template also acts as the base template from which each service repository will extend (via #4) to define the specific trigger scenario and configuration for the service.

The visual representation for this pipeline is shown below, following a linear path from Build to Production.

Sample Pipeline Visualisation

We have defined three stages, each of which references a template to keep the process consistent and promote reuse between deployment stages. Also, note the Sources as outlined showing the pipeline is using resources from two repositories.

 

 

2. Build

This is where the build related actions can be defined within our pipeline. The build stage is defined within the build.yaml template and is used to facilitate the build time guardrails and process.

In our example we include the following tasks:

  • Run tests where Category=UnitTest from projects referenced in the .sln file
  • Build the container
  • Execute container scanning against the build container image and fail if there any vulnerabilities medium or higher.
  • Push the image to container registry if successful.

From the executed pipeline, we can view and discover the unit test results.

Integrated Test Results

Generally the build stage of a pipeline is where tooling such as code / container scanning and robust testing can provide a fast feedback loop for teams.

A full example for the template can be found here.

 

3. Deployment

The use of the deploy-<env>.yaml template approach provides a clean separation for environment specific variables, allowing us to reduce the complexity of the azure-pipeline.ci-cd.yaml template.

We can see from the below code that the template provides a wrapper around the deploy.yaml template to inject environment specific parameters.

The deploy.yaml template contains the tasks that will be executed across all environments. In this scenario, we simply have 1 task to update the intended version of the container.

4. Trigger

Finally, we can see why we've taken this design approach. Each service repository includes a simple pipeline definition that extends from the azure-pipeline.ci-cd.yaml template, providing only the specific parameters that are required. 

The service does not have any explicit knowledge of how the pipeline works, or the steps that are executed, allowing services to be on-boarded quickly and consistently.

Environment Approvals & Checks

Approvals & checks are managed within each defined environment (for example dev, test, production). Here we can apply our checks to ensure  allow-listed templates are being extended from, and define specific approvers before the pipeline stage is executed.

Environment Approvals and Checks

As the approvals and checks are applied at the environment level, different rules can be applied at different environments, such as requiring different approvers for production vs test environments, and not requiring any approvers for lower dev focused environments.

Wrapping Up

We have seen through the codification of guardrails into our Azure DevOps processes and pipelines we can support delivery teams and stakeholder groups while building and releasing in a safe and repeatable way.

Teams and stakeholders should continue to review and evolve the guardrails in place to ensure they remain contemporary and fit for purpose. Approaches, tooling, and best practices will evolve over time and our guardrails must keep pace.

I would love to hear the types of guardrails you have implemented and found to work well as well as those that have not!

 

If you like what you read, join our team as we seek to solve wicked problems within Complex Programs, Process Engineering, Integration, Cloud Platforms, DevOps & more!

 

GET IN TOUCH!

Leave a comment on this blog: