So, you want to cut-over hey…
Releasing your hard-earned code, platforms and new customer experiences can be daunting and challenging but also a very rewarding experience. After all, you have spent your blood, sweat and tears to finally get to a point where your software can be used. Whether it be a daily deployment, a monthly release or a mammoth year-long go-live, each cut-over needs to be intricately planned, prepped and executable for the greatest chance of success.
This blog post is intended to provide release managers, release engineers and service delivery experts comfort that they have considered a range of criteria and are as much prepared for a cut-over as possible. So, without further ado… Here is your top 11 list!
Note: The purpose of this article is not to advise on the complexities of software deployments, rather, everything else around a go-to-market release that should be considered. For consideration of specific software deployment techniques, my esteemed colleagues have written many articles such as:
- GitOps Driven Deployments on OpenShift
- Mule on OpenShift Part 1 – Deployment Models
- Set up automated deployment of Boomi applications using Azure DevOps
- Lessons learnt from deploying Azure Search via CI-CD.
Top 11 Tips!
- The runsheet
- Executor, back-up resource and escalation point
- Automation is key
- Rollback plan
- Time-based checkpoints, communication checkpoints and decision points
- Testing, Testing, Testing!
- Dress Rehearsals
- Entry and Exit Criteria Defined
- Ensure your Command Centre/War Room is in place
- Have Fun!
As a starting point, a visualisation or graphical depiction of the cut-over period is an excellent way to determine the sequence of events required to make a successful cut-over. It is also perhaps one of the most challenging steps. It allows various stakeholders to understand and agree upon the sequence of events (including deployments) and encourages collaboration across teams, which is especially important for a multi-vendor environment. The level of detail required should be relative to the complexity of a release, the example below is a multi-vendor, multi-platform release over a whole weekend.
Figure 1: Example Cut-over plan visualisation
The visualisation of a cut-over is an incredibly useful tool for communicating with team members and senior stakeholders and will provide them with confidence in the ability of your team to release successfully.
2) The runsheet
The most frequently used and well-understood tool for a successful cut-over is a comprehensive runsheet, described simply as a document that lists the action steps to take during a specific process, i.e. Cut-over.
Tips for creating an effective runsheet:
- Ensure there are activities/steps/increments no longer than 30min. If so, break them up into discreet tasks so you can actively track progress and understand blockages.
- The medium or tool used to document a runsheet does not have to be prescriptive: some people prefer Microsoft Project, others Microsoft Excel, or even Atlassian’s Confluence. The key criteria for the runsheet is that it is shareable, can be understood by multiple parties and is version controlled.
- Identify the critical path. This is “the longest sequence of activities from start to finish that must be completed to ensure the cut-over is finished by a certain time”. By understanding the critical path, you gain insight into the most important activities of a runsheet, the timelines and the correlation between tasks. This gives you a greater understanding of which task durations you can modify, and which must stay the same.
- Identify and document dependencies.
- Have multiple reviewers across different disciplines and teams.
Figure 2: Example runsheet
The runsheet is a necessary best-practice asset. By curating and sharing a comprehensive runsheet, it enables you and multiple teams to understand and track the discreet steps required during a cut-over.
3) Executor, back-up resource and escalation point
Each task must have an identified executor, backup team member and escalation point. Most importantly this must be understood and agreed upon by all parties prior to cut-over as it will typically be done after hours. Chances are someone will get sick or incur a snap lockdown. This step allows you to avoid single points of failure and is a key risk-mitigation tool.
4) Automation is key
If you haven’t noticed, we folks here in Core Business Engineering are incredibly passionate about automation. Any steps you can take to automate deployments, testing, in-built security functions, communications and anything else in the runsheet is time well spent. Benefits include:
- Eliminating the chance of anyone making mistakes.
- Reducing the time taken to complete the tasks.
- Enabling a more frequent release of features.
- Freeing up your experts to make improvements to deployment processes, security, and quality of the environment.
5) Rollback plan
Don’t leave this to the last minute, a solid rollback strategy must be in place, documented, tested, and should be included in the runsheet. In addition, the ‘point of no return’ must be identified, understood, and communicated. This is the point in time at which rollback is no longer possible.
A comprehensive rollback plan is an essential risk mitigation tool that allows for you to return the production system or systems back to their pre-cut-over state should cut-over be unsuccessful.
6) Time-based checkpoints, communications checkpoints and decision points
In your visualisation and runsheet, various points in time must be identified as a stocktake and/or decision point to ensure you have completed all essential prior tasks. For each of these checkpoints, there should be an agreed ‘definition of done’. This helps track if the execution of cut-over is being performed successfully.
These checkpoints also serve as a valuable point in time to provide communications on progress to senior stakeholders or any other interested parties. It is important to select and know your who, when, how and why of each of these communications items. This includes your agreed means of communication, be it SMS, email, or virtual meetings.
7) Testing, Testing, Testing!
The most important step of any cut-over is, of course, testing that your newly released code works as expected! Three must-have considerations of cut-over testing are:
- Production Verification Tests and Business Verification Tests – Technical and Business Verification steps must be pre-defined and executable. Make sure you understand and include methods to clean up any residual data from verification testing that you do not want to remain in your production environment.
- Security Testing – Don’t get caught out by security at the last minute. You need to ensure that all your security testing steps are understood, and the success criteria is clearly defined. Will you be penetration testing as a part of cut-over? If so, make sure you understand what level of risk is acceptable by the business prior to cut-over.
- Environment Testing – This is especially important if you are deploying to production for the first time. You don’t want to be caught out without knowing that the systems actually talk to each other. Make sure your DNS Changes are implemented and validated. Don’t forget your private and public keys, and IP whitelisting!
Testing is an essential step of any cut-over to make sure your production system is working as expected.
8) Dress Rehearsals
You must plan and execute a Dress Rehearsal prior to any major release. What this looks like will vary greatly depending on the type of cut-over, what is possible, the environments available and the data available. The information you gain from this process will be invaluable, whether it be to determine the actual time of loads and processes or whether your different platforms and respective teams can successfully communicate with each other.
Dress rehearsals can be categorised into both technical and operational rehearsals:
- Operational rehearsals simulate business processes, IT handoff points and L1-L3 operations to ensure you are ready for go-live and that any learnings are captured and catered for.
- Technical dress rehearsals test the technical processes of a cut-over to refine the activities in runsheet, record timings and identify any technical limitations. It is can also be used to test rollback activities.
Note - Don’t forget to plan for and remove your data if you are doing Dress Rehearsal testing in production!
9) Entry and Exit Criteria Defined
Prior to any cut-over, you need to ensure your entry and exit criteria are defined and agreed on. This is the entry point to a release and determines the success criteria to allow you to enter a cut-over. It should be actively tracked well before any cut-over, so you understand what the shortcomings are prior to any release and whether you should be entering into a cut-over. This is a considerably large topic area and one I will be covering in future blog posts.
10) Ensure your Command Centre/War Room is in place
The Command Centre/War room is where all the action takes place. This may be coordinated virtually, but it is advisable to get as many of the key players in a single area as possible to actively resolve issues. Co-location allows for an easier means of communication, faster resolutions, and less waiting time to make sure someone is online and available! Some key considerations are:
- Location – does it have after-hours access?
- Communications bridge – do you require multiple, or one? Who has access? Is the same bridge being used for communications to senior stakeholders?
- What equipment does everyone involved require? Especially the developers.
- Are one or multiple rooms required? Some break-out areas may be necessary.
- Consider your offshore vs onshore model and what tasks can be executable by which teams. There may be data restrictions for offshore staff, especially when working with sensitive government data.
11) Have Fun!
Cut-overs can be demanding and stressful, but also a fun and exciting time. Make sure you add the little things to make the whole process fun. Some classic ideas to pump up the adrenalin and make those midnight deployments less stressful are:
- Co-locating: bring everyone together as a team to enjoy the moment and get those good vibes going.
- Recognise checkpoints and call out to teams completing their tasks – you can even encourage friendly competition between teams to get the competitive spirit going.
- Have refreshments and food in place, some snacks and pizza can go a long way.
- Create a playlist and get a speaker available, nothing like some good tunes to pump the team up and argue over songs when you are not smashing out tasks (provided it is not too distracting).
- Identify any potential breaking points and be ready to support the teams.
By creating a fun atmosphere, you allow for a greater level of team collaboration, energy, and a more memorable experience!
I promised you a Top 11, but I could go on forever. So, here are some honourable mentions:
- Change Champions – Don’t forget about the change! One important consideration for a cut-over is to make sure you have your change champions and super users identified. They can be the difference between success and failure, after all, perception is everything.
- Change Advisory Board/Technical Advisory Board Approval – Probably a good idea to make sure you have approval before you go live!
- Pre-execution steps defined – Have you done your pre-cutover extracts? Have you paused your batch jobs? I certainly hope you have!
- Retrospective - Post-release retrospectives should be conducted, and actions documented for future releases.
Ensuring a successful cut-over is not an easy job. It requires cross-team collaboration, technical proficiency, and a willing and able team ready to take on and change their organisation. I hope this list has:
- helped you prepare for your next cut-over,
- given you confidence that you are ready to go, or
- scared you into action!