How to choose the right Upgrade Method for your Mule Runtime On Premises

As we’ve seen in this previous post, there are mainly three different methods to upgrade your Mule Runtime:

Each of these methods has its pros and cons. In this other post, we’ve done an overview of each of them. Now the question is - Which one should we choose? Which one is best for our deployment and requirements?

In this post, we will think about what parameters we need to take into consideration to decide which upgrade method is better for us:

Downtime Period

The first thing we need to ask ourselves is how critical are our apps and what’s the impact of having an interruption of the service.
From that analysis we will know two things:

If Zero Downtime is required (we can’t afford stop our runtimes at any time). If that’s the case we’ll have to go for a Full Manual Upgrade
In case Zero Downtime is not required and we can find an upgrade timing window, how long is that period? Based on that we could use the Mule Upgrade Tool or the Upgrade via Runtime Manager

Resources required

Based on the strategy required for the upgrade we might need to provide additional resources during the process, such as extra servers, licenses or load balancers.

An upgrade can be in-place or in a parallel server.

If we run the upgrade in-place, we will be running the upgrade on the same current server, so no extra servers and licenses are required. If an in-place upgrade is possible then we’ll go for the Mule Upgrade Tool.
If we run the upgrade in a parallel server, we’d need to provide an extra server and, eventually, extra licenses and/or a load balancer. In this case, both the Full Manual Upgrade and the Upgrade via Runtime Manager are possible.

This is an important aspect to take into consideration, as extra resources come at extra cost and we need to make sure we can afford them

URLs of our apps

The URL of our apps should not changed after upgrading the Mule Runtime, that would be a really bad experience for the consumers of our apps/APIs. Consider that an upgrade consists of moving between minor versions (from 4.x to 4.y), so this is frequent operation we need to perform for our apps. We don’t want to change the URL of our apps after an upgrade every three months.
With that in mind:

If we perform an in-place upgrade the URLs of our apps will remain the same, because the server remains the same and, hence, the DNS name won’t change. This means we don’t have to worry about the URLs using the Mule Upgrade Tool
If we perform the upgrade using a parallel server this means that our apps will be now running on a different server. That server will have a different DNS name. To solve the problem we need to make sure, before the upgrade, that our current server is behind a Load Balancer or that we’re using a CNAME DNS record pointing to our server. That will allow us to change the Load Balancer or the CNAME and point them to the new server without modifying the URL of our apps

This is something that should have been set up at the moment of initial installation. If your URLs are pointing directly to the hostname of the server then - either you go for an in-place upgrade (using the Mule Upgrade Tool) or you modify the URL of your apps, setting it up correctly for this upgrade and the following ones.

If your server is using a Load Balancer or a CNAME then, setting up a parallel server is an upgrade option for you. In that case you can choose among any of the three upgrade methods.

Testing and Validation

Once the runtime upgrade is done, the testing and validation of our apps in the new mule runtime version is mandatory in any of the three upgrade methods. However, to choose wisely our upgrade method we need to evaluate internally:

If there’s an existing testing strategy prepared for the upgrade. What type of tests you’d be running (unit, integration, regression, performance, ...), what test cases are necessary, how to run them...
How much of that testing is ( or can be) automated

With that, we’d need to estimate how long would it take for us to complete the testing and validation after the upgrade. Knowing how much time we need for the testing is really important, because we need to know, beforehand, if we could complete the testing and validation within the upgrade timing window.
For example:

If we use the Mule Upgrade Tool or the upgrade via Runtime Manager, we’ll be stopping the current runtime during the upgrade. This means all the testing and validation have to be done within the upgrade timing window.
If our testing require more time than the available window then we’ll have to go for a Full Manual Upgrade. The manual upgrade allows us to perform all the tests in parallel, while the current runtime is running.

Lastly, consider that the upgrade timing window is not fully dedicated to testing and validation. There are more tasks to do during the timing window such as:

Stop the current server
Backup of the current server
Replicate configurations
Run the upgrade (in-place or in a parallel server)
Redeployment of all the apps (if necessary)
Testing and Validation
Rollback (if the upgrade is not valid)

Redeployment of apps

As we saw in our previous post, in some upgrade methods we will need to redeploy our apps to the new Mule Runtime version. The in-place upgrade (using the Mule Upgrade Tool) does not require it. The upgrade via Runtime Manager and the Full Manual upgrade do.
So, to understand what upgrade method is best for us, we need to know a couple of things:

Can the (re)deploy process be automated? Do we have an existing CI/CD pipeline for that? If so, how long will it take to deploy all the apps?
If the redeployment process has to be manual, same question, how long will it take to do it manually?

And again, knowing how much time we need for this is very important, because this is something that will have to be done during the upgrade timing window.
With that in mind:

The Mule Upgrade Tool does not require the apps to be redeployed
The upgrade via Runtime Manager and the full manual upgrade require the apps to be redeployed
The upgrade via Runtime Manager requires the runtime to be stopped while we upgrade the runtime and redeploy the apps. The Full Manual Upgrade does not require it, the redeployment of the apps can be done while the current runtime is running
So, if your timing window upgrade is not long enough to perform all the tasks we detailed in the previous section, then you’ll have to go for a Full Manual Upgrade

Automation

We’ve seen it in the previous sections but it’s worth reinforcing the message. Automation plays a key role in upgrade operations. As we’ve mentioned, automation will help us:

In the upgrade of the runtime (Mule Upgrade Tool)
In the (re)deployment of our apps
In the testing and validation after the upgrade

So, evaluate your current automation strategy and see how it can help you with the upgrade process and what could be changed to help you with the upgrades

Rollback Strategy

Last, but not least, another important aspect you need to assess from each upgrade method is its rollback strategy. The three methods provide a rollback strategy:

The Mule Upgrade Tool performs a backup of the current server before upgrading and provides a sub-command to rollback automatically to the previous version
The upgrade via Runtime Manager can be rolled back by stopping the new server and get the old one back up again.
The Full Manual Upgrade just requires to undo the changes in your Load Balancer or CNAME and point them back to the old server again

Knowing that, you need to now:

Test each rollback strategy. NEVER NEVER NEVER go ahead with an upgrade without having tested and knowing what you’d have to do if the upgrade fails
Evaluate how complex each method is for you
Estimate how much time you’d need to rollback an upgrade. This time counts towards your upgrade timing window.