So, you want to cut-over hey…

Releasing your hard-earned code, platforms and new customer experiences can be daunting and challenging but also a very rewarding experience. After all, you have spent your blood, sweat and tears to finally get to a point where your software can be used. Whether it be a daily deployment, a monthly release or a mammoth year-long go-live, each cut-over needs to be intricately planned, prepped and executable for the greatest chance of success.

This blog post is intended to provide release managers, release engineers and service delivery experts comfort that they have considered a range of criteria and are as much prepared for a cut-over as possible. So, without further ado… Here is your top 11 list!

Note: The purpose of this article is not to advise on the complexities of software deployments, rather, everything else around a go-to-market release that should be considered. For consideration of specific software deployment techniques, my esteemed colleagues have written many articles such as:

• GitOps Driven Deployments on OpenShift
• Mule on OpenShift Part 1 – Deployment Models
• Set up automated deployment of Boomi applications using Azure DevOps
• Lessons learnt from deploying Azure Search via CI-CD.

Top 11 Tips!

• Visualisation
• The runsheet
• Executor, back-up resource and escalation point
• Automation is key
• Rollback plan
• Time-based checkpoints, communication checkpoints and decision points
• Testing, Testing, Testing!
• Dress Rehearsals
• Entry and Exit Criteria Defined
• Ensure your Command Centre/War Room is in place
• Have Fun!

1) Visualisation

As a starting point, a visualisation or graphical depiction of the cut-over period is an excellent way to determine the sequence of events required to make a successful cut-over. It is also perhaps one of the most challenging steps. It allows various stakeholders to understand and agree upon the sequence of events (including deployments) and encourages collaboration across teams, which is especially important for a multi-vendor environment. The level of detail required should be relative to the complexity of a release, the example below is a multi-vendor, multi-platform release over a whole weekend.

Figure 1: Example Cut-over plan visualisation

The visualisation of a cut-over is an incredibly useful tool for communicating with team members and senior stakeholders and will provide them with confidence in the ability of your team to release successfully.

2) The runsheet

The most frequently used and well-understood tool for a successful cut-over is a comprehensive runsheet, described simply as a document that lists the action steps to take during a specific process, i.e. Cut-over.

Tips for creating an effective runsheet:

• Ensure there are activities/steps/increments no longer than 30min. If so, break them up into discreet tasks so you can actively track progress and understand blockages.
• The medium or tool used to document a runsheet does not have to be prescriptive: some people prefer Microsoft Project, others Microsoft Excel, or even Atlassian’s Confluence. The key criteria for the runsheet is that it is shareable, can be understood by multiple parties and is version controlled.
• Identify the critical path. This is “the longest sequence of activities from start to finish that must be completed to ensure the cut-over is finished by a certain time”. By understanding the critical path, you gain insight into the most important activities of a runsheet, the timelines and the correlation between tasks. This gives you a greater understanding of which task durations you can modify, and which must stay the same.
• Identify and document dependencies.
• Have multiple reviewers across different disciplines and teams.

Figure 2: Example runsheet

The runsheet is a necessary best-practice asset. By curating and sharing a comprehensive runsheet, it enables you and multiple teams to understand and track the discreet steps required during a cut-over.

3) Executor, back-up resource and escalation point

Each task must have an identified executor, backup team member and escalation point. Most importantly this must be understood and agreed upon by all parties prior to cut-over as it will typically be done after hours. Chances are someone will get sick or incur a snap lockdown. This step allows you to avoid single points of failure and is a key risk-mitigation tool.

4) Automation is key

If you haven’t noticed, we folks here in Core Business Engineering are incredibly passionate about automation. Any steps you can take to automate deployments, testing, in-built security functions, communications and anything else in the runsheet is time well spent. Benefits include:

• Eliminating the chance of anyone making mistakes.
• Reducing the time taken to complete the tasks.
• Enabling a more frequent release of features.
• Freeing up your experts to make improvements to deployment processes, security, and quality of the environment.
  
  

5) Rollback plan

Don’t leave this to the last minute, a solid rollback strategy must be in place, documented, tested, and should be included in the runsheet. In addition, the ‘point of no return’ must be identified, understood, and communicated. This is the point in time at which rollback is no longer possible.

A comprehensive rollback plan is an essential risk mitigation tool that allows for you to return the production system or systems back to their pre-cut-over state should cut-over be unsuccessful.

6) Time-based checkpoints, communications checkpoints and decision points

In your visualisation and runsheet, various points in time must be identified as a stocktake and/or decision point to ensure you have completed all essential prior tasks. For each of these checkpoints, there should be an agreed ‘definition of done’. This helps track if the execution of cut-over is being performed successfully.

These checkpoints also serve as a valuable point in time to provide communications on progress to senior stakeholders or any other interested parties. It is important to select and know your who, when, how and why of each of these communications items. This includes your agreed means of communication, be it SMS, email, or virtual meetings.

7) Testing, Testing, Testing!

The most important step of any cut-over is, of course, testing that your newly released code works as expected! Three must-have considerations of cut-over testing are:

• Production Verification Tests and Business Verification Tests – Technical and Business Verification steps must be pre-defined and executable. Make sure you understand and include methods to clean up any residual data from verification testing that you do not want to remain in your production environment.
• Security Testing – Don’t get caught out by security at the last minute. You need to ensure that all your security testing steps are understood, and the success criteria is clearly defined. Will you be penetration testing as a part of cut-over? If so, make sure you understand what level of risk is acceptable by the business prior to cut-over.
• Environment Testing – This is especially important if you are deploying to production for the first time. You don’t want to be caught out without knowing that the systems actually talk to each other. Make sure your DNS Changes are implemented and validated. Don’t forget your private and public keys, and IP whitelisting!

Testing is an essential step of any cut-over to make sure your production system is working as expected.

8) Dress Rehearsals

You must plan and execute a Dress Rehearsal prior to any major release. What this looks like will vary greatly depending on the type of cut-over, what is possible, the environments available and the data available. The information you gain from this process will be invaluable, whether it be to determine the actual time of loads and processes or whether your different platforms and respective teams can successfully communicate with each other.

Dress rehearsals can be categorised into both technical and operational rehearsals:

• Operational rehearsals simulate business processes, IT handoff points and L1-L3 operations to ensure you are ready for go-live and that any learnings are captured and catered for.
• Technical dress rehearsals test the technical processes of a cut-over to refine the activities in runsheet, record timings and identify any technical limitations. It is can also be used to test rollback activities.

Note - Don’t forget to plan for and remove your data if you are doing Dress Rehearsal testing in production!