The problem
We did an in-place upgrade of an Azure VM from Windows Server 2016 Datacenter to Windows Server 2022. The VM ran IIS (an HTTP/HTTPS site plus a line-of-business app on a custom port). After the upgrade the server never came back:
- RDP and the websites didn’t respond.
- The Azure portal showed the VM as Running, but its provisioning state was stuck at Updating, and the guest agent wasn’t reporting.
- Two restarts and a stop/start (deallocate) didn’t help.
Step 1: Look at the console, not the network
When an Azure VM won’t answer, the first stop is Boot diagnostics. The screenshot showed:
? Working on updates 98%
? Don’t turn off your PC. This will take a while.
? Your PC will restart several times.
Get-AzVMBootDiagnosticsData -ResourceGroupName -Name -Windows -LocalPath C:\Temp\bootdiag
The activity log showed the upgrade had started around 11:30 AM. By evening it had been at 98% for roughly nine hour
Step 3: Read the upgrade logs from a rescue VM
You can’t read the logs on a VM that won’t boot, so we:
- Created a managed disk from the snapshot.
- Built a temporary rescue VM (Server 2022, no public IP, same subnet).
- Attached the disk copy as a data disk.
- Used Azure Run Command to bring the disk online, give it a drive letter, and read the logs. We never had to log in to the rescue VM.
$copy = New-AzDisk -ResourceGroupName -DiskName ‘-OS-repaircopy’ `
-Disk (New-AzDiskConfig -Location -CreateOption Copy -SourceResourceId $snap.Id -SkuName StandardSSD_LRS)
Invoke-AzVMRunCommand -ResourceGroupName -VMName -CommandId RunPowerShellScript -ScriptPath .\inspect.ps1
? Tip: the attached volume may come up without a drive letter. Check Get-Partition and assign one with Set-Partition -NewDriveLetter.
What the disk told us
- The registry on the disk already reported Windows Server 2022 Datacenter, build 20348. The upgrade had switched over to the new OS.
- C:\Windows.old existed, holding the previous 2016 install.
- C:\Windows\Panther\setupact.log showed every upgrade step as executed (migration, security, OOBE, services) except the last one:
Operation (executed): Start suspended services
Operation (remaining): Cleanup Windows.old at end
Executing operation: Cleanup Windows.old at end
CSimpleCleanup::DoExecute: Will remove path: C:\Windows.old\Windows
VERBOSE: Path \?\C:\Windows.old\Windows has no exceptions, trying a straight delete
The log ended there, on every boot.
setuperr.log was full of errors, but they were the usual non-fatal migration noise (plugin load failures, GAC assembly errors and so on), not the cause.
Root cause
The upgrade had actually finished. Setup was hanging on its final cleanup step: deleting C:\Windows.old\Windows. That folder turned out to contain:
? 1,004,868 files in 111,920 folders: 95 GB
Setup deletes this one file at a time on a single thread, during first boot, on a Standard SSD. It wasn’t finishing, and every restart went back into the same step.
Step 4: Delete the folder offline, in parallel
To test the fix without risk, we made a second disk copy from the snapshot and attached it to a second rescue VM. We also temporarily switched that disk to Premium SSD (detach, change SKU, reattach) for higher IOPS during the delete.
Instead of a normal delete, we used robocopy to mirror an empty folder over the target:
$empty = ‘C:\emptydir’
New-Item -ItemType Directory -Force $empty | Out-Null
robocopy $empty ‘G:\Windows.old\Windows’ /MIR /B /XJ /R:0 /W:0 /MT:64 /NFL /NDL /NP /NJH
cmd /c rd /s /q ‘G:\Windows.old\Windows’
??????????????????????????????????????????????????????????????????????????????????????????????????
? Switch ? Why ?
??????????????????????????????????????????????????????????????????????????????????????????????????
? /MIR ? Makes the target match the empty source, which deletes everything in it ?
??????????????????????????????????????????????????????????????????????????????????????????????????
? /B ? Backup mode: gets past TrustedInstaller-owned ACLs without hours of takeown/icacls ?
??????????????????????????????????????????????????????????????????????????????????????????????????
? /XJ ? Skips junction points, so it can’t get caught in a junction loop ?
??????????????????????????????????????????????????????????????????????????????????????????????????
? /MT:64 ? 64 parallel threads instead of Setup’s one ?
??????????????????????????????????????????????????????????????????????????????????????????????????
? /R:0 /W:0 ? No retries or waits on stubborn files ?
??????????????????????????????????????????????????????????????????????????????????????????????????
We ran chkdsk /scan against the volume first to rule out filesystem corruption.
Result: the full 95 GB / 1 million-file tree was gone in 45 minutes.
?