A colleague here at Semperis recently looped me into a conversation with the manager of a large Active Directory environment running on Windows Server 2008 R2. With end of support for Windows Server 2008 and 2008 R2 coming up soon (officially January 14, 2020), planning is well underway for upgrade of the company’s forest and 110 domain controllers to Windows Server 2016 (the end state selected by this particular company). But one component of the upgrade plan is proving to be difficult, and I’ll explain why.
Hope for the best, plan for the worst
In this organization, any significant change to IT infrastructure requires an approved project plan that includes remediation measures in the event something goes wrong. In the case of an upgrade, that means a way to “go back” if the upgrade fails for some reason or proves to be problematic (for example, breaks a mission-critical application). While “going back” is fairly straightforward with many upgrades, it’s not the case with AD.
That’s because an AD upgrade is more than upgrading (or rebuilding) individual DCs: you’re also making changes to each domain and to the entire forest, and at least one of those changes is irreversible. Imagine that the mission-critical application that doesn’t work with the new (and more secure) AD functionality is a handwritten dinosaur app whose developers all retired long ago. In this situation, you may be looking at having to restore AD from backup and running on the older version until the application can be updated or replaced.
Better… but there’s still a gotcha
Historically, upgrading AD required three irreversible changes:
1. Schema: A schema upgrade is required before upgrading the first DC in the forest (or introducing the first up-level DC). Schema changes have always been – and still are – irreversible.
2. Domain functional level (DFL): Once all the DCs in a domain have been upgraded (or demoted out of the environment), the next step in the AD upgrade process is raising the DFL. (An exception is upgrading from 2016 to 2019: there’s no functional level for 2019, so there’s no need – or even possibility – to raise the DFL.)
Historically, raising the DFL was an irreversible change. However, starting with Windows Server 2012, it’s possible to roll back the DFL. There are some caveats, as outlined in Microsoft’s Windows Server 2012 and Windows Server 2016 upgrade guides. But for most organizations upgrading AD from 2008 or 2008 R2, rollback is possible.
3. Forest functional level (FFL): Once all the domains in the forest have been upgraded, the next step in the AD upgrade process is raising the FFL. (Again, there’s an exception if you’re upgrading from 2016 to 2019.)
As with DFL, raising the FFL was historically an irreversible change, but rollback is now possible. (Note: Rollback to 2008 FFL is possible only if the AD Recycle Bin has not been enabled.)
While two of the three “point of no return” steps may now be reversible, upgrading AD still requires an irreversible schema upgrade. It’s that moment when you pause before pressing the key to continue. And if you’re upgrading an AD that you inherited or that’s been around for a while, you might pause a bit longer.
Risk mitigation
If your AD is healthy, upgrading the schema isn’t generally a problem. However, management doesn’t like to hear that there’s no way back. And let’s be honest: Any AD administrator worth the title hesitates before pushing the key to start an irreversible step.
If you do a risk assessment matrix, the risk falls under the category of low probability but high impact, and should therefore have a mitigation plan in place. For an AD upgrade, risk mitigation means forest recovery.
A challenging proposition
Here’s the problem: forest recovery is no simple task. You probably back up DCs regularly, but DC backups aren’t enough – you also need detailed information about your AD topology, as well as a reliable method of recovery. There’s no native tool for forest recovery, and the manual process outlined by Microsoft is very exacting. In my experience, few AD teams have ever attempted a forest recovery, even in a lab environment.
The good news is that third-party tools are available to automate recovery and ensure you have the necessary backups to recover your AD environment. Semperis AD Forest Recovery is one such tool:
Semperis automates forest recovery, thereby providing the required remediation measure for your AD upgrade plan. Semperis’s Anywhere Recovery and IP mapping capabilities also facilitate upgrade testing in the lab prior to the production upgrade.
A permanent safety net
Of course, an upgrade isn’t the only thing that puts your AD at risk. Cyberattacks are a constant threat. For example, a recent article on Wired.com describes how an attacker took out all the DCs for the 2018 Winter Olympics in Seoul, South Korea.
Not all AD recovery tools protect against this type of threat. For example, they may reintroduce malware in system state and bare-metal backups, or struggle to restore to different virtual or physical hardware. So, it’s important to choose a tool that covers cyber scenarios (ransomware, wiper attacks, etc.) and not just operational scenarios (such as schema upgrades or administrative errors, DIT corruption, AD software failures, etc. that were concerns in the early days of AD).
While an AD upgrade might be the impetus (or opportunity) for procuring an AD recovery tool, the right tool can provide value long after the upgrade. This post from Ed Amoroso, cybersecurity expert and former Chief Security Officer at AT&T, is a great place to learn more.
