With the constant evolution of governance, compliance and security, it can be a daunting prospect to balance best practices while promoting lean and efficient delivery approaches.
In this post, we will walk through and demonstrate how using best practices pipeline structures, standardisation, and cross-team collaboration allows us to establish guardrails to support our build and release processes. The codification of these guardrails provides comfort to stakeholders that their obligations are being considered and enforced, and delivery teams can be confident that they are building and deploying in a safe and compliant way.
What is a guardrail? Like the safety barriers that keep us safe when driving, guardrails in a pipeline aim to make the processes we use to build and release safe by ensuring required tasks (technical, governance, compliance or security related) are implemented in an agreed way. Embracing a guardrail approach means teams can build and release with confidence knowing they are aligned to, and supported by the organisational requirements for safe delivery.
Note: There may be scenarios where teams need or want to approach a problem in a different way to that defined in a guardrail. The intent of a guardrail is not to dictate the only approach, but to define an agreed approach. When proposing an alternative approach teams will likely need to seek approval from the respective stakeholder(s); this may be fed back into the agreed guardrail approach, or teams may be asked to align with the existing approach.
Setting the scene
To help give some context to the problem and approach, let’s explore a scenario that we will apply in the proceeding design and discussion.
As part of a new initiative, a series of containerised .NET Core services will be developed and deployed to Azure App Services. Containerisation has been chosen as the packaging approach to provide flexibility for future hosting decisions and promote standardised delivery. Configuration for services will be managed though Azure App Configuration to ensure configuration is externalised from the application logic.
The following is a logical view of the target state Azure resources:
Within the context of the scenario, let’s assume we have had workshops with security, compliance and operational teams, and the following have been identified as core objectives:
- The build and deployment steps for each service must be consistent and centrally defined to allow tasks to be added or removed over time.
- Unit tests must run and pass as part of the build process to drive quality.
- Container scanning must be incorporated, with any detected medium or higher severity results blocking the build / release process.
- To support audit and change management requirements, only specific permitted pipelines should be granted permission to deploy into environments.
If we can establish supporting guardrails for the above objectives, we can provide comfort to our stakeholders, and teams can focus on delivering business value.
Azure DevOps Features
We will be leveraging several Azure DevOps features as we work through the above scenario, each provides us a part of the overall puzzle to establish our guardrails.
| Repository Resource | Allows us to reference multiple repositories within our pipeline to reference shared artefacts. |
| Extending from template | Provides us with the ability to centrally define and manage a set of pipeline definitions and extend from those base templates for each service we develop. This will enable high reuse and low coupling between services and the centralised pipelines. |
| Required Template |
Enables us to apply a check that ensures only pipelines that have extended from allow-listed pipelines are permitted to be deployed. This provides us a level of security that only our agreed base pipeline templates can be used when deploying, ensuring our guardrails are not being bypassed. |
| Approvals | Ensures that specific users or groups have explicitly approved the release into an environment. |
Pipeline Design
Now we make an assumption we are using a multi-repo design for our initiative - with a repository per service and a ‘service-base’ repository to hold shared assets for the pipelines. The same pipeline approaches can be used in both multi-repo and mono-repo designs.
A copy of all pipeline templates and demo service can be found here.
Expanding from the repository structure, the below describes the key templates in our design that will enable the implementation of the guardrails.
1. Orchestration
Orchestration between the build and release stages within the pipeline is managed within the azure-pipeline.ci-cd.yaml template. The template also acts as the base template from which each service repository will extend (via #4) to define the specific trigger scenario and configuration for the service.
The visual representation for this pipeline is shown below, following a linear path from Build to Production.
We have defined three stages, each of which references a template to keep the process consistent and promote reuse between deployment stages. Also, note the Sources as outlined showing the pipeline is using resources from two repositories.
2. Build
This is where the build related actions can be defined within our pipeline. The build stage is defined within the build.yaml template and is used to facilitate the build time guardrails and process.
In our example we include the following tasks:
- Run tests where
Category=UnitTestfrom projects referenced in the .sln file - Build the container
- Execute container scanning against the build container image and fail if there any vulnerabilities medium or higher.
- Push the image to container registry if successful.
From the executed pipeline, we can view and discover the unit test results.
Generally the build stage of a pipeline is where tooling such as code / container scanning and robust testing can provide a fast feedback loop for teams.
A full example for the template can be found here.
3. Deployment
The use of the deploy-<env>.yaml template approach provides a clean separation for environment specific variables, allowing us to reduce the complexity of the azure-pipeline.ci-cd.yaml template.
We can see from the below code that the template provides a wrapper around the deploy.yaml template to inject environment specific parameters.
The deploy.yaml template contains the tasks that will be executed across all environments. In this scenario, we simply have 1 task to update the intended version of the container.
4. Trigger
Finally, we can see why we've taken this design approach. Each service repository includes a simple pipeline definition that extends from the azure-pipeline.ci-cd.yaml template, providing only the specific parameters that are required.
The service does not have any explicit knowledge of how the pipeline works, or the steps that are executed, allowing services to be on-boarded quickly and consistently.
Environment Approvals & Checks
Approvals & checks are managed within each defined environment (for example dev, test, production). Here we can apply our checks to ensure allow-listed templates are being extended from, and define specific approvers before the pipeline stage is executed.
As the approvals and checks are applied at the environment level, different rules can be applied at different environments, such as requiring different approvers for production vs test environments, and not requiring any approvers for lower dev focused environments.
Wrapping Up
We have seen through the codification of guardrails into our Azure DevOps processes and pipelines we can support delivery teams and stakeholder groups while building and releasing in a safe and repeatable way.
Teams and stakeholders should continue to review and evolve the guardrails in place to ensure they remain contemporary and fit for purpose. Approaches, tooling, and best practices will evolve over time and our guardrails must keep pace.
I would love to hear the types of guardrails you have implemented and found to work well as well as those that have not!