A two-phase swing migration with a strict stop-and-report discipline.
Environment (sanitized): a lab/POC Active Directory forest contoso.example on a single KVM/Ubuntu host (NAT bridge 192.0.2.1/24). Two Windows Server 2019 domain controllers, an identity-sync service, and one operator working over remote management with a strict stop-and-report discipline. Timeline: a two-phase swing migration — a clean, verified sequence, then the rename/re-IP identity saga. Names below are placeholders for the real hostnames, accounts, and IPs. No credential values, password policies beyond "the domain minimum length", or storage locations for secrets are included by design.
This case study demonstrates supervised execution, not autonomous reliability — the AI agent supplied speed and breadth, while the human operator held the gate, the snapshot, and the independent read-back. That split, not the tool, is what made the migration survivable.
Reading note. This is deliberately written as a narrative, not a log. The dead ends — the theories that were wrong — are included because they are the most transferable part of the story. Every claim here was cross-checked against the project record; where the record itself was corrected over time, the corrected version is the one told here.
The bottom line, for a skimming reader. This case study is a demonstration of supervised execution, not autonomous reliability. An AI agent drafted every step — but the thing that kept this forest alive was never the agent. It was the human operator: a strict stop-and-report gate before every mutation, a snapshot behind every one of them, and an independent read-back from a second vantage after each. The agent supplied the speed and the breadth; the operator supplied the judgment. That split — speed from the machine, authority from the human — is the transferable point, and it is the one most worth copying. The technical detail below is the evidence for it.
Starting state. One domain, two Windows Server 2019 domain controllers (dc-01 at 192.0.2.2 holding all five FSMO roles, dc-02 at 192.0.2.3), forest and domain functional level at the 2016 generation, DFSR-based SYSVOL, and a dormant identity-sync (ADSync) agent living on the primary DC.
End state. Two Windows Server 2025 domain controllers — dc-03 (192.0.2.4, all five FSMO roles + global catalog) and the rebuilt second DC (192.0.2.2, global catalog; this box reclaimed the original primary DC's identity after a clean rebuild — see §5). Forest and domain functional level raised to Windows 2025. Identity sync moved off the DCs to a dedicated member server, member-sync-01 (192.0.2.6). The two 2019 DCs were demoted, powered off, and retained (not deleted) as rollback material.
Duration. The clean part of the job — pre-flight, schema prep, two new DCs, FSMO transfer, sync-service move, decommission of the old DCs, functional-level raise — was a straightforward sequence of verified steps. The second half, the rename-and-re-IP phase and the DC4 identity saga, produced the project's most valuable lessons.
The one-place summary for a reader who will not read the rest. Each is framed as a general principle, not a project-specific directive:
whenCreated/objectGUID after every promotion (§5, §6.3).Full detail: §6 ("What I'd do differently") carries the project-specific reasoning behind each; the numbered items above map to it.
Five rules governed every state change, from the first step to the final rebuild:
snapshot-revert state restriction and a snapshot-became-live-disk trap) are cataloged in §4.Verified read-only before any change was authorized: replication healthy (0/5 failures both directions, all partitions), DFSR running on both DCs (a hard prerequisite — the old FRS was absent), functional levels at the 2016 generation, all five FSMO roles on dc-01, time sync healthy, storage headroom confirmed. Baseline snapshots were taken on both 2019 DCs — the first of the snapshot-everything discipline.
Two things to note from this phase:
repadmin /replsummary showed a lone 110 (enumeration timeout) line. That is normal repadmin query noise in a healthy forest, not a replication failure. It was flagged early because it kept looking alarming later.wbadmin resolved to the GUI applet (wbadmin.msc) in a non-interactive remote session — the backup feature was disabled on both DCs and the engine binary was absent. Decision: rely on the hypervisor disk snapshots (restorable, satisfies the checkpoint rule) rather than enable a backup engine that would require reboots.Run on the schema master with the 2025 media mounted, as a service account that had been (reversibly) added to Schema Admins.
echo C | adprep.exe /forestprep). Result: RC=0, schema advanced three versions (baseline objectVersion = 88 → 91), verified on both DCs, replication still 0/5.dMD), producing an apparently abnormal reading. The schema NC head legitimately has objectClass = dMD — standard, expected. The correct, authoritative check (objectVersion on the schema NC head) returned a clean baseline consistent with the 2016 functional level. No change to AD was needed; the "anomaly" was a measurement error. Lesson: verify an abnormal AD reading with a second, independent method before acting./gpprep run standalone returns RC=1 — it must be paired with /domainprep. The corrected combined invocation returned RC=0 (idempotent skip). Phase 1 closed fully verified.Provisioning. Each new DC was a peer of the working 2019 boxes: 2 vCPU / 8 GB, same bridge, 60 GB disk. The unattended install failed in two distinct, evidence-diagnosed ways before succeeding:
First-boot networking failure, a self-inflicted regression, and a clean revert. The first fresh install booted to a healthy lock screen but never got an IPv4 address (full-capture evidence: zero DHCPv4 DISCOVER packets). The wrong first fix attempt — merging a static IP into the Windows registry offline, onto two guessed adapter GUIDs — produced a corrupt TCP/IP profile that crashed the guest on network-stack init. Reverting surfaced a hypervisor limitation: snapshot-revert refused the disk-snapshot state outright ("Invalid target domain state 'disk-snapshot'. Refusing"), so the revert was done manually — destroy the VM, move the bad thin overlay aside for forensics, re-point the domain's system disk at the clean base image, re-define, start. The box recovered; the operator then set the IP inside the guest, where it belonged. Lesson recorded: for a fresh install with an unmapped NIC, in-guest configuration is faster and safer than offline registry surgery.
Domain join. Verified from the other DC's vantage (the reliable one): the AD computer object showed the new FQDN and service principal names — fields written only by a successful join. Multiple "join is broken" signals (empty Domain=, token-cache errors, blank secure-channel reads) turned out to be artifacts of probing inside a SYSTEM-privileged remote session, not a failed join — cross-checked and dismissed.
Promotion blockers, in the order hit (full index in §4): cmdlet parameters that no longer exist on the 2025 build; the DSRM password having to satisfy the domain password policy (the domain's hard maximum caps the length — a too-short candidate was rejected as "too short" and an overly long one as "too long", which is how the cap was discovered); a UTF-16LE-encoded script that cmd.exe cannot parse (silent no-op — the process started and never ran); a blocking console-read over a stdin-less remote session; rapid remote-session churn producing HTTP 500s; a stale-file false positive (the "file exists" check was validating an abandoned earlier variant — the fix was to hash-verify the exact file before launch); and finally the real wall — a credential realm mismatch: the promotion scripts had been building the domain-qualified account name with a wrong realm spelling, which makes the KDC reject the ticket (error 1825) and DC discovery fail (error 1908). Network, DNS, secure channel, and privileges all looked fine — the exact signature of a realm mismatch. Correcting the realm (or using the unambiguous UPN form) was the operator's diagnosis, and with it the promotion completed.
Result. First 2025 DC promoted, verified (replication 0/5 both directions, global catalog, all core services running), and — per the one-at-a-time rule — work stopped for sign-off before the second box. The second DC followed the same proven path. Two benign DNS-delegation warnings on each promotion (expected in a root forest with no parent zone) were recorded and closed.
With both 2025 DCs promoted, replicating, and one holding the global catalog, all five FSMO roles were transferred to dc-03 (the new primary) — operator-authorized, then independently verified from both DC vantages that all five roles sat on dc-03 and that replication remained 0/5. Notably, one earlier cleanup attempt on the second DC (an identity scrub) had been reverted by a reboot — the reboot itself proved the root cause (a registry-level identity source re-painting the old name on every registration), which became a key lesson and a recurring theme of §5.
The dormant ADSync agent on the primary 2019 DC had to come off the DC (sync services belong on a member server, and the DC was about to be demoted). This phase had its own failure cluster:
Forest and domain functional level raised to Windows 2025 — the irreversible final step, gated on both live DCs being 2025.
The WinRM lesson: Set-ADForestMode/Set-ADDomainMode are hang-prone mutating AD writes — the interactive confirmation over a stdin-less remote stream blocks, and every client-side timeout leaves an orphaned remote PowerShell on the DC holding an AD write lock; those orphans stack, and eventually jam even read-only AD queries. The operator therefore ran both raises from the interactive console, where they completed cleanly on the first try — no orphans, no lock-jam. Independent verification from both DC vantages (functional levels read back, replication 0/5, FSMO stable, sync service healthy, all core services running) closed the entire 2019→2025 migration.
Rule carried forward: long-running AD DS writes belong on a console/RDP session, not over a remote channel with short client timeouts. And the completion gate is always the live read-back, never the command's success line.
Goal: give the second new 2025 box the original dc-01 identity (192.0.2.2) and settle the naming. Operator-confirmed method: netdom computername (the supported path for a DC identity change — not Rename-Computer, which doesn't handle DC identity/SPN/DNS), then the IP change, each preceded by a snapshot and a stop-and-report gate.
This phase is where the project got hard, and where the record is most honest about itself. In order:
undefine/define cycle refused to start with a misleading "Failed to get write lock" error. The first working theory — an orphaned kernel file lock — was tested against evidence and formally withdrawn (a libvirtd restart did not clear it; a full device/inode/lock scan found no lock holder anywhere; a non-blocking lock test acquired the "locked" file successfully as the hypervisor user). The ground truth was the kernel audit log: AppArmor was denying a plain read of the VM's base disk, because the per-domain AppArmor profile for this one VM included a .files fragment that did not exist — the profile body was stale (its mtime predated the whole crisis), and the VM's XML disk-chain listing was truncated, so the allowlist did not cover the real backing chain. The fix was surgical and scoped: rebuild the XML to the full disk chain, create the missing .files fragment mirroring a healthy VM's, scoped-reload that one profile only (never a global teardown), then start and fully verify — including confirming the two unaffected VMs stayed healthy before and after. Zero AppArmor denials after the fix.undefine/define can end up with stale security-policy artifacts that a normal re-define does not regenerate.3. The IP migration, done right the second time. The first in-guest IP swap failed catastrophically (removed the management IP before the new one was confirmed — the session died and the auto-revert could not run, leaving the NIC with no address at all). The lesson: add the new IP alongside the old, verify reachability through a second session, then remove the old. The second attempt, using that dual-IP method, succeeded.
DemoteDC4 scheduled-task landmine — discovered incidentally, diagnosed, disabled, and removed, with all credential-bearing scripts scrubbed (see §4).Each entry: symptom → root cause → how it was diagnosed → fix.
4.1 adprep prompt hang (forestprep). Symptom: adprep /forestprep hung at 0% CPU with no output. Root cause: the tool's interactive "type C to continue" confirmation; over a remote session with no interactive stdin, the blocking read never returns. Diagnosis: process state (alive, idle, no output) + knowledge that the prompt exists; confirmed by the same hang class recurring elsewhere in the project (promotion prompts, FL-raise confirmations). Fix: pipe the answer into the process (echo C | adprep.exe …). Same family, same fix: promotion prompts, and (for the FL raise) the decision to run the command from a console instead (§3.7).
4.2 /gpPrep syntax error. Symptom: standalone /gpprep → RC=1, "Invalid syntax. /gpprep must be used with the /domainprep switch." Root cause: the switch is only valid combined with /domainprep. Fix: adprep.exe /domainprep /gpprep → RC=0. (Trivial in hindsight; recorded because it cost a cycle.)
4.3 Schema "anomaly" false alarm. Symptom: a pre-check reported a non-standard objectClass and an apparently abnormal schema state. Root cause: self-inflicted — the check queried a non-existent attribute and mis-read the schema NC head's class; objectClass = dMD on that head object is standard and expected. Diagnosis: re-queried with the authoritative method (objectVersion on the schema NC head) → clean baseline (88), consistent with the 2016 functional level; class objects verified standard. Fix: none — no AD change was needed. Lesson: verify an "abnormal" AD reading with a second independent method before acting.
4.4 DSRM password policy wall + parameter changes on Server 2025. Symptom A: promotion rejected one DSRM candidate as "too short" and another as "too long" — both with the same generic error. Root cause A: the DSRM must satisfy the domain password policy, whose maximum length is hard-capped on this domain — so the double rejection was the diagnostic signature of a policy length cap, not two different failures. Fix A: use a DSRM that satisfies the domain policy. (Re-encountered again during the final rebuild: the below-policy candidate was rejected, the policy-compliant one accepted.) Symptom B: Install-ADDSDomainController on the 2025 build threw "parameter cannot be found" for parameters that exist in 2019-era docs. Root cause B: the 2025 cmdlet's parameter set changed. Fix B: probe the live parameter set (Get-Command … -ParameterList) and build the invocation from evidence. The asymmetry bites twice: Install-ADDSDomainController takes -SafeModeAdministratorPassword, while Uninstall-ADDSDomainController on 2025 takes -LocalAdministratorPassword and no longer has -SafeModeAdministratorPassword. Also: the Windows feature name for AD DS on 2025 is AD-Domain-Services (not the legacy name), and Install-WindowsFeature has no -NoReboot parameter on this build.
4.5 The DC4 identity saga. Told in full in §5. In one line: stale service-principal names for the box's old name kept reappearing within a minute of every clean removal, across three demote/re-promote cycles — until the record's decisive evidence (a reused, pre-existing damaged AD object) reframed the whole problem.
4.6 Hypervisor "write lock" failure that was actually AppArmor. Symptom: virsh start on one VM failed with "Failed to get write lock"; a libvirtd restart did not help. Root cause: AppArmor denial — the VM's per-domain profile included a .files fragment that did not exist on disk, and the domain XML under-listed the disk backing chain; the kernel audit log showed a plain read denied on the base disk file. Diagnosis: the misleading error string led first to a (wrong) orphaned-lock theory, formally withdrawn after lock-hunting found nothing. The ground truth was the kernel audit record: profile name, file, denied_mask="r". Cross-referenced against a healthy VM's intact profile artifacts. Fix: rebuild the XML to the full backing chain; create the missing .files fragment mirroring the healthy VM's; scoped-reload only that one profile (no global teardown); start + verify; confirm the two unaffected VMs stayed healthy. Zero denials after. Lesson: trust the kernel audit log over the error text; undefine/define cycles can leave stale security-policy artifacts that a re-define does not regenerate.
4.7 The hypervisor snapshot-revert limitation (two encounters). Symptom: virsh snapshot-revert refused: "Invalid target domain state 'disk-snapshot'. Refusing." Root cause: the VM was in a disk-snapshot state that revert does not accept. Workaround: manual revert — destroy the VM, move the bad thin overlay aside (preserved for forensics), re-point the domain's system disk at the clean base image, re-define, start. Second encounter (later, worse): a "revert" to a snapshot had actually captured the post-mutation state — the snapshot had become the live disk (the snapshot-became-live-disk trap) — so the true pre-mutation image was the backing file. blockrevert/snapshot-revert were both unavailable; recovery was a backing-chain swap built with assertions against a self-referential chain (the first attempt failed with a lock error; the fixed builder asserted no self-reference before re-define). Lesson: before reverting, verify the snapshot's actual contents, not its name.
4.8 DNS forwarder and the 6900 loop. Symptom: during the identity saga, the identity-sync engine on member-sync-01 threw a storm of sync-failure event 6900s (~29 events in a 20-minute window), and AD-side lookups on the sync account looked disturbed. Working theory: the DCs' DNS forwarders had been lost, breaking external-name resolution needed by the sync engine. What the evidence actually showed: a same-day earlier read showed forwarders present on all three machines; a later read returned empty — an unresolved contradiction that was never re-verified. What did close: after the identity issues settled, no new 6900s appeared for 12+ hours and the engine cycled normally (its own healthy-cycle events present), so the 6900 storm was confirmed as a consequence of the identity/AD-side disturbance, not an independent DNS failure. The forwarder contradiction was left explicitly unresolved in the record. Honest status for publication: the 6900 loop is closed (no recurrence; engine healthy). The "forwarders lost" hypothesis is unverified — it was recorded as an open contradiction, never confirmed or refuted. Per the tasking, this is marked unverified rather than stated as fact.
4.9 The DemoteDC4 scheduled-task landmine. Symptom: an incidental read-only finding: a scheduled task on the renamed box, named for an old box, that ran a one-shot demotion script. Root cause (forensics): a leftover from an earlier automation attempt — created by the service account, running as SYSTEM, pointing at a script that invoked Uninstall-ADDSDomainController with a hardcoded plaintext credential. Its scheduled trigger window had already passed (last run: failed, result -1), but the task was still enabled and allowed on-demand firing — "Next Run Time: N/A" is not safety. Exposure: a manual/on-demand fire (or a future scheduler fix) would silently demote a live DC; the script also sat on the DC with embedded credentials. Remediation (authorized, staged): (1) disable the task (reversible) and verify state read-back; (2) unregister the task and delete all credential-bearing one-shot scripts from the DC, each verified gone; (3) confirm the DC still healthy afterward. Standing rules produced: a DC must never carry a task/script that can self-demote — especially in a small domain; Enabled=true + AllowStartOnDemand=true is armed even with a past-due schedule; never hardcode plaintext credentials in one-shot scripts on a DC.
4.10 The FL-raise WinRM lesson. Symptom (from earlier attempts): Set-ADForestMode/Set-ADDomainMode over the remote channel blocked on the interactive confirmation; each client timeout left an orphaned remote PowerShell on the DC holding an AD write lock; the orphans stacked and eventually jammed even read-only AD queries. Root cause: long-running mutating AD writes over a stdin-less remote session with short client timeouts = orphan buildup + lock contention. Fix: run these writes from an interactive console; the two FL raises then completed cleanly on the first try, with no orphans and no lock-jam. Verification: live read-back from both DC vantages. General rule: AD DS mutating writes that confirm interactively belong on a console; never trust the success line — the gate is always the read-back.
4.11 False-success incidents (two, both root-caused).
OK/ERR) is the only ground truth, never the prose summary — and always reconcile with an independent post-state count.4.12 Other cataloged items (for completeness). virsh reboot is a no-op on these guests (verify reboots by uptime/last-boot time, never domstate); msiexec /x hung on the sync agent's packages four times (switch to direct removal once a "standard" tool keeps wedging); QEMU VNC is single-client (a stray client black-screens noVNC — kill the stale one); repadmin "110/1" lines are query-timeout noise, not replication failures; a rollback script's sed pattern silently no-oped against the hypervisor's XML tag format (caught by an end-state assertion; fixed with a first-occurrence-only replace plus post-write verification).
The symptom, in one sentence. After the renamed box took on the original identity, its AD object kept re-accumulating the old name's service-principal names — 15 stale SPNs reappearing within 60 seconds of every clean, verified removal, across three separate demote/re-promote cycles, plus a stale DNS record and a stale DFSR topology object. The domain was, at points, effectively running on a single DC, or on one healthy DC plus a zombie whose AD-side identity had been deleted.
Why this section is honest about the timeline: the wrong theories were pursued across multiple sessions, before the actual root cause was read out of the directory itself. The sequence below is the order the theories were actually tried and ruled out.
Tcpip\Parameters hostname keys feeding DNS self-registration — every registration re-painted the old FQDN and re-added the factory SPNs. That fix worked at the time. In the later crises, the same key was fixed again, the LSA cache was rebuilt after reboot, the SPNs were cleanly removed — and the exact same 15 re-appeared within 60 seconds. Ruled out: while the registry key was a real identity source, it was not the writer of the recurring pattern.CachedMachineNames. Deleted and rebuilt (the stale old-name entry was present, fixed, verified). The re-addition continued. Ruled out.CN=Computers container instead of OU=Domain Controllers. Fixed and proven (15/15 clean removal) — and the SPNs still came back. Ruled out as the persistent cause.The actual root cause — found by reading the directory's own metadata, not by hunting processes:
CN=DC-B,OU=Domain Controllers: created well before the final promotion — with a fixed object GUID, low historical USN.NTDS Settings) was freshly created at the final promotion.Install-ADDSDomainController was reusing the pre-existing, damaged computer object — the scar tissue of every prior crisis lived on the AD object itself — and attaching a brand-new DSA to it. The box's local copy of that object carried the old SPN list at promotion time and held the higher USN, so it re-asserted the stale SPNs and won every DRS merge against scrubs performed from the other DC (lower USN can never win). This single fact explains why a "fresh install" reproduced the identical symptom across three demote/re-promote cycles, and why no local-state cleanup could ever stick.The fix, in three verified steps: (1) demote (the 2025 parameter set, again); (2) delete the entire object family — the DC object, its DSA, the stale DFSR topology member, and the fourth orphan — each verified GONE from the authoritative DC's view, with a domain-wide sweep confirming zero stale-name SPNs remained; (3) rebuild the VM from scratch and re-promote against the now-empty name — which forced the promotion to mint a brand-new AD object. Acceptance check that proves it was genuinely new: whenCreated = today, and a different object GUID than the old one. The post-promotion stability watch (multiple samples, both vantages) held clean: zero stale SPNs, stable count, replication 0/5 both directions, FSMO intact on the primary, global catalog active. The saga closed cleanly.
What this costs in hindsight: at least three full demote/re-promote cycles, two single-DC exposure windows, one zombie DC, one credential lockout, and a long stretch of the project's remaining calendar — all because the fix had to start from "which AD object is this box actually running on?" instead of "which process is writing?" The single most valuable move in the whole saga was the day someone stopped diagnosing the box and started reading the object's creation date and GUID.
netdom /ADD or even a fresh build). The swing migration is the same work; the rename phase is pure risk.Install-ADDSDomainController. Two read-only checks cost nothing and would have saved the project considerable effort: read whenCreated and objectGUID of the resulting OU=Domain Controllers object after promotion. A whenCreated older than the promotion means a damaged object was reused and the box must be demoted, its object family deleted, and re-promoted.DemoteDC4 task (armed, enabled, hardcoded credential, past-due schedule) was a live gun in the domain. Standing rule: no Uninstall-ADDSDomainController-capable task or script may exist on a DC; check Enabled/AllowStartOnDemand, not just "Next Run Time."| Item | State |
|---|---|
| Domain | contoso.example, 2 live DCs |
| DC A (primary) | Windows Server 2025, 192.0.2.4, all 5 FSMO roles, global catalog, replication 0-fail both vantages |
| DC B | Windows Server 2025, 192.0.2.2 (original primary identity), global catalog, fresh AD object (new GUID, today's creation date), 0 stale SPNs, autostart enabled |
| Identity sync | member-sync-01 @ 192.0.2.6, engine live and healthy, timezone aligned to the DCs |
| Functional level | Windows 2025 forest + domain (irreversible step complete, both vantages verified) |
| Old 2019 DCs | dc-01 and dc-02 demoted, powered off, retained (disk images + all snapshots as rollback material) |
| Hypervisor snapshots | 34+ retained per policy (baseline, pre-forestprep, pre-domainprep, pre-join, pre-promo, pre-demote, pre-flraise, pre-rename, pre-ipchange, pre-demote-rebuild, and others) |
| Lab note | This was a lab/POC environment — single hypervisor host, evaluation OS editions, single-domain forest. Findings on cmdlet parameter changes, AD-object reuse, AppArmor/libvirt interaction, and the discipline itself transfer directly to production forests; the credential-handling notes deliberately do not. |
GroupPolicy 1125 errors — flagged for investigation; the sync engine itself is healthy (live, cycling normally), so this is cosmetic-or-unknown, not confirmed harmful. (Unverified in the record — marked as such.)Get-DnsServerForwarders on both DCs before dismissing. (Unverified in the record — marked as such.)For a reader with no project context, the pattern worth copying is not any single fix but the operating loop that repeated ~40 times in this project:
snapshot → GO → execute → independent read-back (authoritative vantage)
→ time-gapped re-read → report (including what still fails)Prepared from the full project record (DCupgrade-STATE.md, MIGRATION-LOG.md, and the working case-study entries). No state-changing commands were run to produce this document. Retracted claims from earlier in the project — the "orphaned flock" theory, the AD Connect writeback theory, "dual-DC restored" statements that later proved to describe a zombie, and the "15 SPNs removed" reports that removed zero — are not carried forward as fact; they appear above only as the diagnosed dead ends they were.