Platform Engineering

Andrew Rawson

11 essential items for a cut-over

Posted by Andrew Rawson on 13 July 2021

release, release management, cut-over, runsheet, command centre, rollback, dress rehearsal, cut-over plan, go-live

So, you want to cut-over hey…

cut over

Releasing your hard-earned code, platforms and new customer experiences can be daunting and challenging but also a very rewarding experience. After all, you have spent your blood, sweat and tears to finally get to a point where your software can be used. Whether it be a daily deployment, a monthly release or a mammoth year-long go-live, each cut-over needs to be intricately planned, prepped and executable for the greatest chance of success.

This blog post is intended to provide release managers, release engineers and service delivery experts comfort that they have considered a range of criteria and are as much prepared for a cut-over as possible. So, without further ado… Here is your top 11 list!

Note: The purpose of this article is not to advise on the complexities of software deployments, rather, everything else around a go-to-market release that should be considered. For consideration of specific software deployment techniques, my esteemed colleagues have written many articles such as:

Top 11 Tips!

 

1) Visualisation

As a starting point, a visualisation or graphical depiction of the cut-over period is an excellent way to determine the sequence of events required to make a successful cut-over. It is also perhaps one of the most challenging steps. It allows various stakeholders to understand and agree upon the sequence of events (including deployments) and encourages collaboration across teams, which is especially important for a multi-vendor environment. The level of detail required should be relative to the complexity of a release, the example below is a multi-vendor, multi-platform release over a whole weekend.

Visualisation Example

Figure 1: Example Cut-over plan visualisation

 

The visualisation of a cut-over is an incredibly useful tool for communicating with team members and senior stakeholders and will provide them with confidence in the ability of your team to release successfully.

 

2) The runsheet

The most frequently used and well-understood tool for a successful cut-over is a comprehensive runsheet, described simply as a document that lists the action steps to take during a specific process, i.e. Cut-over.

Tips for creating an effective runsheet:

  • Ensure there are activities/steps/increments no longer than 30min. If so, break them up into discreet tasks so you can actively track progress and understand blockages.
  • The medium or tool used to document a runsheet does not have to be prescriptive: some people prefer Microsoft Project, others Microsoft Excel, or even Atlassian’s Confluence. The key criteria for the runsheet is that it is shareable, can be understood by multiple parties and is version controlled.
  • Identify the critical path. This is “the longest sequence of activities from start to finish that must be completed to ensure the cut-over is finished by a certain time”. By understanding the critical path, you gain insight into the most important activities of a runsheet, the timelines and the correlation between tasks. This gives you a greater understanding of which task durations you can modify, and which must stay the same.
  • Identify and document dependencies.
  • Have multiple reviewers across different disciplines and teams.

 

Runsheet Example

Figure 2: Example runsheet

The runsheet is a necessary best-practice asset. By curating and sharing a comprehensive runsheet, it enables you and multiple teams to understand and track the discreet steps required during a cut-over.

 

3) Executor, back-up resource and escalation point

Each task must have an identified executor, backup team member and escalation point. Most importantly this must be understood and agreed upon by all parties prior to cut-over as it will typically be done after hours. Chances are someone will get sick or incur a snap lockdown. This step allows you to avoid single points of failure and is a key risk-mitigation tool.

 

4) Automation is key

If you haven’t noticed, we folks here in Core Business Engineering are incredibly passionate about automation. Any steps you can take to automate deployments, testing, in-built security functions, communications and anything else in the runsheet is time well spent. Benefits include:

  • Eliminating the chance of anyone making mistakes.
  • Reducing the time taken to complete the tasks.
  • Enabling a more frequent release of features.
  • Freeing up your experts to make improvements to deployment processes, security, and quality of the environment.

5) Rollback plan

Don’t leave this to the last minute, a solid rollback strategy must be in place, documented, tested, and should be included in the runsheet. In addition, the ‘point of no return’ must be identified, understood, and communicated. This is the point in time at which rollback is no longer possible.

A comprehensive rollback plan is an essential risk mitigation tool that allows for you to return the production system or systems back to their pre-cut-over state should cut-over be unsuccessful.

6) Time-based checkpoints, communications checkpoints and decision points

In your visualisation and runsheet, various points in time must be identified as a stocktake and/or decision point to ensure you have completed all essential prior tasks. For each of these checkpoints, there should be an agreed ‘definition of done’. This helps track if the execution of cut-over is being performed successfully.

These checkpoints also serve as a valuable point in time to provide communications on progress to senior stakeholders or any other interested parties. It is important to select and know your who, when, how and why of each of these communications items. This includes your agreed means of communication, be it SMS, email, or virtual meetings.

Checkpoint

 

7) Testing, Testing, Testing!

The most important step of any cut-over is, of course, testing that your newly released code works as expected! Three must-have considerations of cut-over testing are:

  • Production Verification Tests and Business Verification Tests Technical and Business Verification steps must be pre-defined and executable. Make sure you understand and include methods to clean up any residual data from verification testing that you do not want to remain in your production environment.
  • Security Testing – Don’t get caught out by security at the last minute. You need to ensure that all your security testing steps are understood, and the success criteria is clearly defined. Will you be penetration testing as a part of cut-over? If so, make sure you understand what level of risk is acceptable by the business prior to cut-over.
  • Environment Testing – This is especially important if you are deploying to production for the first time. You don’t want to be caught out without knowing that the systems actually talk to each other. Make sure your DNS Changes are implemented and validated. Don’t forget your private and public keys, and IP whitelisting!

Testing is an essential step of any cut-over to make sure your production system is working as expected.

8) Dress Rehearsals

You must plan and execute a Dress Rehearsal prior to any major release. What this looks like will vary greatly depending on the type of cut-over, what is possible, the environments available and the data available. The information you gain from this process will be invaluable, whether it be to determine the actual time of loads and processes or whether your different platforms and respective teams can successfully communicate with each other.

Dress rehearsals can be categorised into both technical and operational rehearsals:

  • Operational rehearsals simulate business processes, IT handoff points and L1-L3 operations to ensure you are ready for go-live and that any learnings are captured and catered for.
  • Technical dress rehearsals test the technical processes of a cut-over to refine the activities in runsheet, record timings and identify any technical limitations. It is can also be used to test rollback activities.

Note - Don’t forget to plan for and remove your data if you are doing Dress Rehearsal testing in production!

Rehearsal

9) Entry and Exit Criteria Defined

Prior to any cut-over, you need to ensure your entry and exit criteria are defined and agreed on. This is the entry point to a release and determines the success criteria to allow you to enter a cut-over. It should be actively tracked well before any cut-over, so you understand what the shortcomings are prior to any release and whether you should be entering into a cut-over. This is a considerably large topic area and one I will be covering in future blog posts.

10) Ensure your Command Centre/War Room is in place

The Command Centre/War room is where all the action takes place. This may be coordinated virtually, but it is advisable to get as many of the key players in a single area as possible to actively resolve issues. Co-location allows for an easier means of communication, faster resolutions, and less waiting time to make sure someone is online and available! Some key considerations are:

  • Location – does it have after-hours access?
  • Communications bridge – do you require multiple, or one? Who has access? Is the same bridge being used for communications to senior stakeholders?
  • What equipment does everyone involved require? Especially the developers.
  • Are one or multiple rooms required? Some break-out areas may be necessary.
  • Consider your offshore vs onshore model and what tasks can be executable by which teams. There may be data restrictions for offshore staff, especially when working with sensitive government data.

11) Have Fun!

Cut-overs can be demanding and stressful, but also a fun and exciting time. Make sure you add the little things to make the whole process fun. Some classic ideas to pump up the adrenalin and make those midnight deployments less stressful are:

  • Co-locating: bring everyone together as a team to enjoy the moment and get those good vibes going.
  • Recognise checkpoints and call out to teams completing their tasks – you can even encourage friendly competition between teams to get the competitive spirit going.
  • Have refreshments and food in place, some snacks and pizza can go a long way.
  • Create a playlist and get a speaker available, nothing like some good tunes to pump the team up and argue over songs when you are not smashing out tasks (provided it is not too distracting).
  • Identify any potential breaking points and be ready to support the teams.

By creating a fun atmosphere, you allow for a greater level of team collaboration, energy, and a more memorable experience!

Have fun

 

I promised you a Top 11, but I could go on forever. So, here are some honourable mentions:

  • Change Champions – Don’t forget about the change! One important consideration for a cut-over is to make sure you have your change champions and super users identified. They can be the difference between success and failure, after all, perception is everything.
  • Change Advisory Board/Technical Advisory Board Approval – Probably a good idea to make sure you have approval before you go live!
  • Pre-execution steps defined – Have you done your pre-cutover extracts? Have you paused your batch jobs? I certainly hope you have!
  • Retrospective - Post-release retrospectives should be conducted, and actions documented for future releases.

Conclusion

Ensuring a successful cut-over is not an easy job. It requires cross-team collaboration, technical proficiency, and a willing and able team ready to take on and change their organisation. I hope this list has:

  1. helped you prepare for your next cut-over,
  2. given you confidence that you are ready to go, or
  3. scared you into action!

 

 

If you like what you read, join our team as we seek to solve wicked problems within Complex Programs, Process Engineering, Integration, Cloud Platforms, DevOps & more!

 

GET IN TOUCH!

Leave a comment on this blog:

You might also enjoy: