When Deloitte Platform Engineering teams start on a new client, we bring with us our development processes that help deliver value.

These processes continuously improve as they are applied to many, many engagements and evolve as trends and industry consensus change. This means that we can have all the maturity that comes with establishing a new development team that normally takes many months, in a matter of days.

One of the first questions I am often asked is "what branching strategy should we use?". This is a valid question but, in my opinion, hardly the first one we should answer.

The way we work and the way we release products define the branching strategy. Not the other way around.

It is to my eternal frustration when defining release management processes is deferred till much later in the project, because "production release is months away". And when the time comes, so does the rapid realisation that the branching strategy chosen without foresight somehow does not match up with the release management aspirations.

I find it equally frustrating when teams try to strive for excruciatingly generic processes, rather than spending the time gathering requirements and understanding what is needed. If we are building, say a set of microservices, then why not design a process that makes our most frequent use case as frictionless as possible?

Keeping the microservices theme, in this post, I'd like to lay out what I consider to be an optimal approach to branching strategies, CI/CD and release (which I collectively call "Development Workflow") when delivering microservices.

For the impatient, you can jump down to The Patterns chapter to see what this looks like. The intervening chapters will discuss the why before we get to the how. And as a developer, I can tell you that I will not buy into process unless I know why I am doing it.

Parameters

Designing processes in a vacuum is a good way of coming up with something truly terrible. So instead let us define some guardrails to give us focus as we solution.

For this, I have come up with an imaginary new engagement and defined some hypothetical requirements. 

In addition to those requirements, there are also some less imaginary best practices, that I normally like to see in one such process:

As the years have gone by, it is becoming rarer that I need to argue about some of the above points. What was once considered controversial is now common wisdom. That said, I would like to expand on a few topics below.

On Microservices Architecture

We have written extensively about microservices in this blog so I will keep things light here. In the context of this post, the following characteristics of a microservices architecture are relevant:

• Domain decomposition, which results in multiple services as independently deployable units with runtime dependencies to each other and downstream systems
• Decentralisation of data and governance along domain lines
• Extensive automation to flatten the overhead of dealing with many, many microservices

On Multiple Git Repos

Mono-repos and multi-repos are both perfectly valid options depending on what you are trying to do.

In a mono-repo setup, all applications are located in the same Git repository. We find this useful when we would want to:

• Treat the entire set of applications as a baseline to be versioned and deployed together
• Give more weight to build-time dependencies, over runtime ones
• Use CI/CD tooling and practices that can support mono-repos well
• Know that building the entire repo does not take an excessively long time
• Have developers who are Git savvy enough to handle frequent merge conflicts and other Git gymnastics

Things get ugly when you try to use mono-repos and treat each application separately, as preferred by microservices architecture. CI/CD needs to map from one repo to many pipelines. It is not plain what versions of each application exist in a given commit, let alone trying to tag the commit with versions. Having to examine the Git diff to understand the incoming changes gets old very fast.

Looking back at our requirements above, one repo per application makes much more sense. The main branch in each repo will track a single application unambiguously. Git tags can signal clearly what version exists within a commit.

Another oft-forgotten benefit of multiple repos is that changes are spread across them. This reduces how many developers are working within a single repo at the same time. There is less chance of complicated merge conflicts and it is rare for developers needing to reach for the more advanced Git commands.

I should note I have encountered some misconceptions when promoting multiple repos in organisations with a legacy development environment. We have to remember we are talking to developers, who have been scarred with a lot of menial and manual tasks in their career and normally in a single repo. To them trying to multiply this out to many repositories would seem daunting, if not impossible.

My approach is to remind them that things have drastically changed and improved in the industry. Much of the heavy lifting has been moved to CI/CD and automation. Adding each additional repository barely moves the needle when it comes to overheads. And there are certainly fewer headaches than when 20 people are working in the same repository.

On Shorter Releases

I hope it is quite intuitive why we prefer smaller and more frequent releases.

When the releases are larger and less frequent, each production release carries a much greater risk. The deployments take longer and require larger deployment windows. There are too many moving parts and triaging issues need to cover a lot of ground. Not to mention, the feedback loop to the QA team, business and end-users, is much longer, which will make no one happy. It is then no surprise that continuous delivery and DevOps principles have gained such foothold in modern engineering practices.

It should be noted that frequent releases cannot be achieved off the bat. As an organisation matures, it develops higher levels of quality assurance, automation and streamlined release management which unlocks this capability. 

We normally treat the size of a release as a dial, turning it down as the maturity increases.

Path to Production

Understanding the CI/CD process is essential in determining how code changes should be delivered.

Targeting a runtime environment, such as Kubernetes, means that the output of our build process is a container image. This is a logical hand-off point between the build and deployment process. By versioning the resulting container image, we can use it to deploy to each environment thus, satisfying "build once, deploy many times" practice.

Most modern processes, like above, use different specialised tools to fulfil each stage of the pipeline. This is a far cry from the old school processes where SCM is dragged (kicking and screaming), throughout the workflow and used to track or trigger environment deployments, etc.

Versioning

An important part of this overall developer workflow is the way an application is versioned.

In the case of microservices, they naturally provide interfaces, such as APIs. These are then consumed by other microservices or upstream systems. This makes consumers an important component of managing a microservice's interface and driving some of the decisions made when versioning that interface.

Here are some of the versioning schemes that you may encounter in the wild:

Static versioning uses a mutable version, which may sound blasphemous. However, it is ideal for those situations where the consumers are happy to be upgraded to the latest version without their consent.

Incremental versioning provides slightly more information as you can distinguish different versions but you cannot glean any more than that.

Temporal versioning is a nice fit for organisations with seasonal releases. IntelliJ IDEA is an example of this where they do quarterly major releases. But similar to incremental versioning, the versions do not provide a whole lot of information about what is contained within.

Semantic versioning is where things get interesting. It can unambiguously signal the nature of changes between two different versions. This allows consumers to make an informed decision on whether they can readily upgrade to the new version or if additional work is required.

In the fast-paced world of modern engineering, it is common to create a new version following each successful build, treating each as a release candidate. While these applications move through environments, bugs may be encountered so further builds and versions will occur. Commonly, some of these intervening versions will never make it to production.

This is a perfectly fine scheme for projects where most of the APIs are internal. What is important is comparing the existing production versions with the incoming one to understand the breadth of incoming changes. 

However, this may be less desired for public APIs, where consumers may wonder what the heck happened to a specific version. Imagine if Spring released v10.0.0 when the last public release was v5.5.3. For these use cases, it may be desired to hold the changes and only release the whole lot to production when ready.

Pre-release extension to semantic versioning makes this possible. Each application can have one or more pre-release stages, indicated by a number or a codename, such as alpha/beta. Given that minor and patch versions are already backward compatible, pre-releases are most commonly used for major versions.

Maven veterans may now be feeling some twangs of familiarity here. SNAPSHOT versions are an example of pre-release versioning.

It should be obvious that there is slightly more planning and effort involved in pre-releases so this needs to be taken into consideration when deciding between the two schemes. That said, you are free to pre-release any version, including fixes, if it can be justified.

If you are curious about how you would implement semantic versioning in an automated manner, you may find my other post, Semantic Versioning using Conventional Commits, of interest.

The Patterns

Let's finally talk about how we are going to do our branching. A strategy is nothing but a set of patterns that can be applied repeatedly. The fewer patterns you have, the more elegant the resulting process.

Here are the patterns that we will apply to our branching strategy. Hopefully, many of these look very familiar.

Mainline Branching

One or more branches are used as the stable, source of truth for the repository. All other branches diverge and later converge back to these. It is also called Trunk Branching, a hangover from the SVN days.

In our workflow, we consider each commit on the mainline branch a release candidate. It means the commit should have gone through the quality gates that we have managed to "shift left". There will be more quality gates later on as the artifact, created from the build, is promoted through environments but we expect the code to be as healthy as we can measure at build time.

master is the traditional name if you have a single mainline branch but we will probably see a more inclusive name surfaces in the coming years.

Feature Branching

To satisfy the parallel development of features, each developer creates a branch from the mainline where they add their changes. Later, these features are introduced back into the mainline by merging (integrating) the feature branches.

Let's play through a scenario where this pattern is applied three times, possibly by multiple developers.

Feature branching is a misnomer as this pattern can be used to essentially deliver any changes to the mainline, be it features, fixes, refactors, etc. These branches are considered short-living and should be deleted upon merge.

The branch naming conventions should be decided by the team, based on their ways of working, and used consistently from then on. Some common conventions include feature/*, feat/*, fix/*, refactor/*, etc. Being short-living, the branch names of less significance than the mainline but should be unambiguous during their lifetime.

On Merging Strategies

As a team, we can choose and enforce a merging strategy, to ensure the Git history looks consistent across the repositories.

Git provides a few commands and flags that can be combined to produce a large combination of merging strategies. Here are some of the major ones to keep in mind:

Squash takes all the changes from the feature branch and collapsed them into a single new commit which is placed on top of the mainline branch. Any history from the feature branch is lost and mainline becomes a series of single-commit features. In rare scenarios where developers are branching off each other's feature branches, rather than waiting for the feature branch to be merged into mainline, this strategy will cause more problems.

Explicit merge commit strategy ensures there is always a commit that denotes the merge into the mainline. This makes the Git history more complex with more superfluous commits but also provides as much information about the code history as possible.

On most occasions, a feature branch contains some initial commits and subsequent commits that address peer review comments (unless you practice amending commits on feature branches). These additional commits are not particularly interesting on their own.

Therefore, I tend to prefer the strategies that create a single commit on the mainline. These keep the history clean, each commit corresponding to a feature or a fix. This also means reverting a change is essentially reverting a single commit.

I should note that the above is only relevant for feature branch patterns. I do not recommend squashing commits when merging between mainline branches as that history is significant and should be preserved.