Rolling back your TDS deployments (and Sitecore Upgrades)

Have you ever had the need to rollback your Sitecore deployments? Or maybe an upgrade went wrong and you need to rollback the changes?

Well you should’ve taken proper backups then, shouldn’t you! Learn your lesson yet? Now go back and add proper back-up steps to your deploy process… Good. Jeez. Shortest blog post ever! Moving on…

The problem with backups is it takes time, which seems to exponentially increase as your Sitecore database increased in size, and an equally heap load of time when you need to restore that database. The other problem with the backup is that it is just a point of time restore, what about changes that have happened since that backup had taken place?

In traditional software development, our deployments would always contain rollback scripts. These would be create ahead of time and be part of the deployment script. If something went wrong, we could rollback without having to do a whole database restore. The issue with Sitecore is that our deployments do not modify the databases directly, instead we are deploying packages of meta-data, a.k.a. update packages.

If we were using the zip type of Sitecore Update Packages then it would be possible to create an anti-package using Sitecore PowerShell Extensions and we could build that into our deploy process.

However, since we are using TDS this generates the .update type packages, and unfortunately like an 🐘 running away from a 🎪 once that baby has bolted, there’s no way of getting it back in – you can’t create an anti-packages of .update packages. It’s not just TDS that uses .update packages though – Sitecore themselves use them for upgrading between versions!

Update Package History

Wait whaaaaaaat? Since when? Since forever apparently! I looked back at Sitecore.Update.dll from v6.5 and the rollback generation was there. So was the code below. Best. Kept. Secret. EVER!

When you install an .update package, Sitecore will backup items and files that it modifies to /temp/UpgradeHistory/Upgrade/ folder which (by default) is located in your website root folder.

You’ll find several files in here, including installation logs, messages generated during installation (i.e. stuff you’d see in the Update Installation Wizard screens) and also a rollback.rlb file. That’s the jackpot!

Yes, these are generated even during automated installation of TDS packages using, for example, Sitecore Ship i.e. not just manually installing packages.

What does an .rlb file contain?

No idea, it’s not a zip file with a package.zip inside. Let’s convert it to a proper Sitecore package instead and then we can look inside…

Post Deploy Step using PowerShell

I’ve previously written about invoking PowerShell scripts from your TDS packages using Post Deploy steps. Using the exact same process, add a script to /sitecore/system/Modules/PowerShell/Script Library/MyCustomProject/Event Handlers/packageinstall/tds/postdeploy to run the following:

	$tempFolder = [Sitecore.Update.Utils.FileUtils]::InstallationHistoryRoot
	$historyFolder = [Sitecore.MainUtil]::MapPath($tempFolder)
	#get the latest update folder
	$latestUpdate = Get-ChildItem -Path $historyFolder | Sort-Object LastAccessTime -Descending | Select-Object -First 1
	Write-Host $latestUpdate.FullName
	[Sitecore.Update.Engine.PackageGenerator]::ConvertRollbackPackage( `
	$latestUpdate.FullName + "\rollbackPackage.rlb", `
	$latestUpdate.FullName + "\rollbackPackage.update")
	Write-Log "[TDSPostDeploy::SPE] Generated rollback package to $($latestUpdate.FullName)"

view raw

Generate Anti-Update Rollback.ps1

hosted with ❤ by GitHub

(If you don’t want to deploy a package to test it just yet, just run the above in the ISE)

The above will generate the anti-update rollback package for the latest deployment in the same folder as Sitecore stores the installation history. Tweak as required to meet your needs (e.g. we save to a different folder and use the version number of our latest deployment).

Yep, it really is as simple as that – a self-generating anti-update package every time you deploy.

Post Deploy Step using C#

If for some reason you have not discovered the magic of Sitecore PowerShell Extensions, then go ahead and go old school with some C#:

	using System.ComponentModel;
	using System.IO;
	using System.Linq;
	using System.Xml.Linq;
	using HedgehogDevelopment.SitecoreProject.PackageInstallPostProcessor.Contracts;
	using Sitecore.Diagnostics;
	namespace ForwardSlash.TDS.PostDeploy
	{
	[Description("Generate anti-update rollback package for latest deployment.")]
	public class GenerateRollback : IPostDeployAction
	{
	public void RunPostDeployAction(XDocument deployedItems, IPostDeployActionHost host, string parameter)
	{
	string tempFolder = Sitecore.Update.Utils.FileUtils.InstallationHistoryRoot;
	string historyFolder = Sitecore.MainUtil.MapPath(tempFolder);
	//get the latest update folder
	var latestFolder = new DirectoryInfo(historyFolder).GetDirectories()
	.OrderByDescending(d => d.LastWriteTimeUtc).First();
	Sitecore.Update.Engine.PackageGenerator.ConvertRollbackPackage(
	latestFolder.FullName + "\\rollbackPackage.rlb",
	latestFolder.FullName + "\\rollbackPackage.update");
	Log.Info("[TDS Post-Deploy] :: Generated rollback file for deployment to " + latestFolder.FullName, this);
	}
	}
	}

view raw

TDS Post-Deploy Anti-Update Rollback Package.cs

hosted with ❤ by GitHub

What’s in the anti-update package?

The update package contains all the necessary information to rollback your instance to the pre-deployment state. This include both items and code – that’s all files including config, views, javascript, dlls. Everything.

As Sitecore touches an item or a file during installation, it throws it into the rollback.rlb file.

Needless to say, it will only generate what’s in the initial .update file you deployed. If you don’t generate TDS packages with items and code (and use some other process for deployment) then it can’t magic that out of thin air. But you took backups of those files before a deployment as well, right? Also bear in mind, if there is a Post Deploy Step that modifies files or items (which a lot of Sitecore Updates do), they also will not included either.

They see me rollin’… back…

Need to rollback? Simply install the .update package using the Update Installation Wizard (/sitecore/admin/updateinstallationwizard.aspx). Or build something in your CI to do it.

So what’s the big deal you say? I already told you to take backups before deploying right? Just roll that back!

When you’re implementing Blue/Green deployments you realise that’s it’s extremely difficult to get the CM server working this way as well. It’s really really tricky. It’s much simpler if you can leave the CM server out of it and implement a content freeze whilst you deploy Blue/Green on the CD server and run whatever tests you need to.

If your tests take more than a few minutes then that will annoy your editors. Or you do this stuff out of hours. I don’t like doing that and then hanging around “just in case”.

Instead, you deploy, allow the content editors to continue with their work after a slight downtime due to app pool recycle after deployment. Now if something is found to be wrong with the deployment, rollback the CM server and don’t flip the Green/Blue switch on load balancer. Since you’re not restoring a database backup, your editors don’t lose any content edits they’ve made in the meantime – obviously, be aware that if their changes were for new templates or new fields in templates, they will be lost. (This is a very simplistic overview of a more complex problem/set up. Maybe I’ll provide details of this in a future post.)

This is no substitute for proper backups. Always take backups before deployments or upgrades. It will save your 🍑

Now go forth… and then rollback ¯_(ツ)_/¯

Big thanks to Steve McGill for helping me figure this stuff out.

Further Reading:

• TDS Post-Deploy PowerShell Step
• What are Post Deploy Steps
• BlueGreenDeployment by Martin Fowler
• Blue/Green deployment strategy in the new TfL website
• Using Blue-Green Deployment to Reduce Downtime and Risk