Building a Secure AKS Employee Portal Backend on Azure with an AI Agent

1. Objective & scope

What was built. A small, cost-conscious AKS cluster in Azure that hosts an internal employee portal. The deployment doubles as a teaching case study — the bar is clear, well-documented, standard patterns over clever configuration, and every design choice is explainable to a learner.

Why. The project exercises three PSC lines simultaneously:

• D-IT — cloud infrastructure / platform engineering (AKS, VNet, NSG, ACR, Key Vault, SQL, Log Analytics).
• R-mgmt — the audit-then-deploy, cost-cap, evidence-first discipline that governed the whole engagement.
• U-training — the case-study narrative itself is the deliverable.

Locked scope (do not re-open without explicit approval).

• Employee-portal only. Internal, no public exposure. No customer / B2C path.
• No VPN gateway. The portal is reached over the VNet / internal LB. A live site-to-site tunnel is not required to demonstrate any of the PoC's actual goals.
• Infrastructure + identity plumbing only. End-to-end application login was intentionally not exercised — the ILB backend pool is empty by design, and the portal application layer is out of scope for this PoC. This is a scope boundary, not an oversight (see §6).
• Budget cap: $300 / month with alerts at 50 / 80 / 100 % to the budget-alert recipient.

What was removed from the original two-portal brief (public customer + internal employee) when the scope was narrowed to employee-only:

• App Gateway v2 + OWASP WAF
• Public IP (for inbound)
• Public-ingress subnet
• Public NSG
• Any B2C / public-identity path

Two optional add-ons still ship as separate templates, applied only when needed — not in the default deploy:

2. Architecture

2.1 Resource topology

Two diagrams, both generated directly from the authoritative Bicep source (main.bicep + cluster.bicep) for this case study. (See §10 for a note on diagram provenance.)

Figure 1 — Network & compute topology (eastus, single resource group cpcs-rg).

Figure 2 — Identity & secrets flow (the "no stored secrets" design).

2.2 Component inventory

2.3 Two-file template split (required by this Bicep build)

Bicep cannot mix subscription-scoped resources (the Resource Group) with resourceGroup-scoped resources (AKS, VNet, …) in one file (BCP135). So:

All three compile clean to ARM JSON with Bicep 0.46.1 (via az bicep).

3. Design decisions & tradeoffs

Each major fork in the road, the decision, and the reason.

3.1 Two-portal → employee-only scope narrowing

Decision. Narrow the original two-portal brief (public customer + internal employee) to employee-only.

Why. The PoC's actual goals — AKS, RBAC, Key Vault, ACR, SQL, Azure AD auth — are all internal. A customer-facing path (App Gateway v2 + OWASP WAF, public IP, public-ingress subnet, public NSG, B2C) was not required to demonstrate any of them, and it was the largest source of cost and complexity. Removing it:

• Cut the largest cost line (~$142 / mo for the gateway, §3.3)
• Removed the recurring East US SKU failure source
• Left the design strictly internal (no public IP at all — a strict improvement over the gateway design)

What changed in the templates. App Gateway v2 + OWASP WAF, public IP, public-ingress subnet, public NSG, and any B2C / public-identity path removed from main.bicep / cluster.bicep.

3.2 Entra External ID considered and ruled out

Decision. Not used. No customer population means no External ID requirement.

Why. The portal is employee-only; the existing <organization-domain> Entra tenant already provides the OIDC issuer and the RBAC model (aadProfile managed=true, enableAzureRBAC=true). Adding External ID / B2C would have introduced a second identity platform, a separate tenant boundary, and a separate set of RBAC grants — none of which the PoC needs.

Note for future readers: Azure AD B2C is closed to new customers as of May 2025. Entra External ID (the successor) is the supported path going forward. If a future PoC does need a customer / B2C path, budget for External ID onboarding, not B2C.

3.3 VPN Gateway dropped from PoC scope

Decision (b, 2026-08-25). Drop the VPN Gateway. No Central US gateway test, no East US quota support ticket.

Why.

• The employee portal is already internal / VNet-reached via the Internal LB; none of the PoC's goals (AKS, RBAC, Key Vault, ACR, SQL, Azure AD auth) require a live site-to-site tunnel.
• The gateway was the largest cost line (~$142 / mo) and the repeated source of East US InvalidGatewaySkuSpecified failures on VpnGw1 / VpnGw2 Standard (see §4.2).
• The design therefore has no public IP at all (pure internal path) — a strict improvement over the gateway design.

Temporary admin/office access during the PoC window is provided by the scoped allow-office NSG rule (<office-source-cidr> → 80/443), not a VPN. That rule is PoC-only — not a production access pattern — and is removed with the rest of the PoC at teardown.

What changed in the templates. gw-basic.bicep / gw-test.bicep / gw-v2.bicep (+ compiled .json) are out of the active deploy path (not referenced by main.bicep / cluster.bicep); kept in the repo only as historical test-harness evidence. GatewaySubnet resource, gatewaySubnetCidr / gatewayLocation / vpnSku params, and the gateway + gateway public-IP resources removed. vpnSourceCidr param renamed officeSourceCidr (<office-source-cidr>) — it now feeds the allow-office NSG rule.

3.4 Cosmos DB considered and ruled out in favor of Azure SQL Database Serverless

Decision. Use Azure SQL Database Serverless v1 (centralus, single database employeedb, schemas for the portal's relational data), not Cosmos DB.

Why.

• Relational data model. The portal's data is inherently relational (employees, departments, access records). Cosmos DB's document model would have forced a denormalized design and a different query language for data that is naturally tabular.
• Simpler cost model. SQL Serverless with minCapacity=1 vCore + autoPauseDelay=3600 s bounds the cost to ~$20–150 / mo and auto-pauses when idle. Cosmos DB's RU-based pricing is harder to reason about for a PoC and has a different floor.
• Simpler identity. One SQL Server Contributor role on the shared MI is straightforward. Cosmos DB would have required a separate RBAC model (Cosmos DB Account Reader / Data Contributor) and a different set of connection-string mechanics.

3.5 Bicep two-file split (subscription-scope entrypoint + resourceGroup-scope module)

Decision. Split the template into main.bicep (subscription scope) + cluster.bicep (resourceGroup scope), with an explicit scope: rg on the module reference.

Why. Bicep cannot mix subscription-scoped resources (the Resource Group) and resourceGroup-scoped resources (AKS, VNet, …) in one file — it emits BCP135. And this Bicep build requires an explicit scope on a module that crosses from a subscription-scoped file into a resourceGroup-scoped target — BCP035 / BCP134. The two-file split is the only clean way to satisfy both constraints.

4. What went wrong, and how it was resolved

This is the most valuable section for a teaching case study. Do not sanitize it.

4.1 Parallel subnet writes → AnotherOperationInProgress / Conflict (5 failed deploys)

Symptom. After the gateway was dropped, deploys failed with AnotherOperationInProgress / Conflict on Microsoft.Network — and the failing subnet changed each run (aks-node, then internal-ingress, then aks-node again).

Root cause. Three subnets of one VNet were being created concurrently, which Azure's Network service rejects non-deterministically.

Fix. Serialize subnet creation with a dependsOn chain: vnet → nodeSubnet → internalSubnet → systemSubnet, with AKS and Key Vault created after the subnets. Also fixed: the agent pool vnetSubnetID originally pointed at the system subnet while the KV ACL + SQL firewall are scoped to aks-node — re-pointed at nodeSubnet so pods can actually reach KV / SQL.

Lesson. Provision subnets serially or with explicit dependsOn from the start. Do not rely on Bicep's implicit dependency ordering for resources that the underlying service rejects in parallel.

4.2 East US VPN Gateway / SQL SKU-availability restrictions

Symptom. Two separate East US restrictions on this subscription:

1. VPN Gateway: InvalidGatewaySkuSpecified on VpnGw1 / VpnGw2 Standard in East US.
2. SQL Server: "Provisioning is restricted in this region" for new SQL servers in East US.

Resolution.

• Gateway: dropped from scope entirely (decision b, §3.3). No longer a blocker.
• SQL: relocated to centralus (approved by user, "2 a"). Consequence: publicNetworkAccess must be Enabled (VNet service endpoints are same-region only) — documented as a PoC-only tradeoff; tighten the firewall before any production use.

Lesson. Check regional SKU availability before designing around a specific gateway or SQL SKU. Query az aks get-versions / az sql servers list-skus / az network vpn-gateway list-skus for the target region and subscription before committing a design.

4.3 Azure Budget ARM 404 for this tenant

Symptom. Microsoft.Resources/budgets returns 404 at both subscription and tenant scope for this tenant (provider is Registered, but the resourceType is not exposed — verified at sub AND tenant scope, api-versions 2024-03-01 / 2022-09-01).

Resolution. Create the budget manually in the Portal after deploy:

Subscriptions → Azure subscription 1 Upgrade → Cost Management + Billing
  → Budgets → New → $300 / Month, alerts at 50/80/100% to <alert-recipient>

The budget params are kept in main.bicep so the Portal budget can reference the same values, but the budget object itself is not created by IaC.

Lesson. Budget-as-code is not universally available. Verify Microsoft.Resources/budgets is exposed in the target tenant before relying on it; have a Portal fallback.

4.4 AKS version / VM-size SKU restrictions in East US

Symptom. Two East US restrictions on this subscription:

1. K8sVersionNotSupported: 1.31.3 is now LTS-only in East US (only 1.31.13 / 1.31.100 LTS patches are GA).
2. VM size not allowed: Standard_D2s_v5 is not in the allowed-VM-sizes list for this subscription in East US.

Resolution. Bumped to 1.34.10 (from az aks get-versions, on the GA / KubernetesOfficial line) and Standard_D2s_v7 (same 2 vCPU / 8 GiB shape as D2s_v5 — same cost band).

Lesson. Pin AKS version + VM size from a live az aks get-versions query, not from a static list. The GA line moves; the LTS line is narrower than it looks.

4.5 Container Insights DCR / DCRA binding failure (AAD → API-server propagation lag)

Symptom. Container Insights is not bound to the Log Analytics workspace. All control-plane indicators confirm this:

• addonProfiles: null on the cluster
• Zero DCRs and zero DCRA associations
• The workspace cpcs-log exists (PerGB2018) but has no data

Two REST PATCH attempts to set logAnalyticsWorkspaceId + the omsagent addon returned HTTP 200 but the fields remained null after 30 s — the API accepted the body silently without applying the binding (likely a 2024-05-01 API limitation where logAnalyticsWorkspaceId is a deprecated top-level property and the correct path requires the omsagent addon + config.logAnalyticsWorkspaceResourceID, which this tenant's API version did not honor).

A separate finding, surfaced while trying to verify the cluster from inside: a temporary AAD cluster-admin grant was registered on the cluster (aadProfile.adminGroupObjectIDs updated, HTTP 200) for the <deployment-service-principal> SP (object ID <deployment-service-principal-object-id>). Despite ~15 minutes of wait + repeated fresh-token attempts, the API server kept rejecting: User "<deployment-service-principal-object-id>" cannot list resource "services" … User does not have access to the resource in Azure — an AAD → API-server propagation / staleness issue outside our control. The grant was revoked and verified (adminGroupObjectIDs: null after az aks update --aad-admin-group-object-ids "").

Resolution (approval 2, 2026-08-26). Accepted as a documented limitation — no further fix attempts (no Portal path, no further DCR / DCRA iterations). The evidence-backed record of this gap is itself a real, evidence-backed finding. No further action is required on this item before publishing.

Lesson. AAD RBAC grants on AKS can lag the API server by 15+ minutes even after the aadProfile is updated and a fresh token is minted. If you're verifying cluster access right after a grant, either wait longer or verify at the AAD layer (role assignment) rather than the API-server layer. And the API can return HTTP 200 on a PATCH that does nothing — always re-read the field you tried to set.

4.6 Post-outage AD replication anomaly (DC1 frozen, DC2 green)

Context. A separate, on-prem AD / DC environment (DC1 = DC1 <dc-internal-ip>, DC2 = DC2 <dc-internal-ip>, both Windows Server 2019, libvirt VMs on a homelab host) suffered a power outage on 2026-08-26. After the host rebooted (06:43 CDT), DC1's inbound-from-DC2 replication state was frozen at the outage window while DC2's own view was green. DC2 was restored from KVM (autostart enabled on both VMs), and the dead PoC neighbor (<poc-vm>, an Azure-DC PoC VM that had been torn down the previous night) was still emitting error 1722 (RPC server unavailable) on 32 consecutive syncs on DC1 and 11 on DC2.

Decision path.

1. Passive vs. forced sync. Chose passive first: let the KDC / LDAP services settle, verify all 7 AD ports (389 / 88 / 53 / 445 / 135 / 3268 / 5985) open on both DCs, then run a scoped, bounded repadmin /syncall DC1 /A (DC2-only-inbound variant) with a 15-minute hard cap. Did not do a forced /replsync or any authoritative state change.
2. Orphaned repadmin processes. A prior full /syncall had hung for 73+ minutes on DC1, leaving 2 orphaned repadmin.exe processes (PID 1184, PID 4360) blocked on LDAP timeouts to the dead neighbor. Decision: kill only those two PIDs (client-side sync helpers — not lsass.exe or ntds), confirm no other repadmin is running, and do not open a second WinRM connection mid-sync. When the targeted kill script ran, the orphans had already self-terminated (their LDAP timeout cycles expired) — the guard correctly found count 0 and did not execute taskkill. Zero repadmin processes confirmed.
3. Scoped sync result. SCOPED_RUN_EXIT=0, completed in ~90 s (well under the cap). Before / after (raw, both DCs):
   • Before (pre-sync, 07:50): DC1 → DC2 5/5 fails (1722) on DomainDnsZones / ForestDnsZones; DC1 inbound-neighbor <poc-vm> 32 consecutive fails.
   • After (post-sync, 17:11): DC1 → DC2 0/5 fails, delta < 30 m, all 5 partitions Last attempt was successful; DC2 → DC1 0/5 fails. replsummary: DC2 42m:31s 0/5 0; DC1 49m:35s 0/5 0.
   • The 58 - <poc-vm> line in replsummary's "operational errors" section remained until the dead-neighbor metadata was removed (§8.2).
4. Final clean resolution. The dead PoC neighbor was fully removed (connection objects + DSA tree + DNS + computer / server / DFSR metadata — see §8.2), and after that removal the error 58 line disappeared and the forest-wide replsummary was clean. No metadata cleanup, USN rollback, or authoritative state change was ever required or performed — the scoped sync + targeted dead-neighbor removal resolved everything cleanly.

Lesson. A frozen replication state on one DC while the other is green is usually a client-side / transport problem (orphaned sync helpers, a dead neighbor, or a post-reboot service not yet listening), not a directory corruption. Do not reach for USN rollback or authoritative restore first — verify the services, clear the orphans, run a bounded scoped sync, and only escalate to metadata surgery if the scoped sync still shows failures on a live neighbor.

5. Security & governance practices demonstrated

5.1 Evidence-first verification discipline

Every claim in this case study is backed by live command output, not a template-success assumption. Concretely:

• Public IP outbound-only (Check 1): verified via az network lb rule list / az network lb inbound-nat-rule list / az network lb outbound-rule list on the node RG — 0 inbound, 0 NAT, 1 outbound (aksOutboundRule). Not assumed from the Bicep.
• NSG effective rules (Check 2): verified via az network nsg rule list + az network vnet subnet show on internal-ingress — p100 / p110 / p4096 declared and attached. The nodepool NSG was verified to have 0 rules (empty by design). The aks-node subnet was verified to have no NSG attached (documented finding, §5.5).
• RBAC granted (pre-deploy): verified via a live az role assignment list on the subscription before the deploy was accepted as ready.
• RBAC revoked (teardown): verified via a live az role assignment list after the Contributor deletion — the row was gone, not merely "accepted".
• Container Insights (Check 3): verified via az aks show --query addonProfiles + az monitor data-collection rule list + a live Kusto query — all three confirmed unbound.
• Portal OIDC (Check 4): verified via az acr repository list (empty) + a project-wide grep for kind: Deployment / kind: Service (none) + az aks nodepool list (inventory) — confirmed no app was deployed.
• Teardown (Check 7): verified via az group show on both RGs (both ResourceGroupNotFound) + az keyvault list ([]) + az role assignment list (only the ABAC-locked UAM row remained).

The discipline: never assume a deploy / deletion / grant succeeded because the CLI returned exit 0. Re-read the state from the API.

5.2 RBAC granted / verified / revoked lifecycle for <deployment-service-principal>

Lesson. A service principal with User Access Administrator cannot revoke its own UAM grant. The holder of the UAM role is locked out of the delete path by the ABAC condition on the assignment itself. Plan for an external Owner / Administrator to perform the final revocation, or disable the SP (az ad sp update --disabled=true) as an immediate, reversible stopgap.

5.3 No-secrets-in-code discipline

Principle. Secrets live in Key Vault, wired via Managed Identity. No client secrets, no K8s-native secret objects, no admin keys.

• Key Vault: <kv-name> (standard, soft-delete 90 d, purge protection ON, network ACL scoped to the two subnets). Pods access via Key Vault CSI driver + the shared MI — no client secrets, no K8s-native secret objects.
• ACR: adminUserEnabled=false, anonymousPullEnabled=false. AKS pulls via the shared MI (AcrPull role). No admin keys ever issued.
• SQL: admin password is a file on the operator's workstation (<sql-admin-password-file>, mode 600) — never in the Bicep, never in the repo, never in the write-up. The Bicep takes it as a param and passes it to the administratorLoginPassword field; the value is supplied at deploy time from the secure file.
• SSH key: public key only, wired into cluster.bicep as ssh.publicKeys[0].keyData. The private key is stored securely outside version control (operator workstation).
• SP credentials: <deployment-service-principal> creds live in a secure file on the operator's workstation (referenced by path only, never by value).

Lesson. The test of a no-secrets design is not "is there a Key Vault?" — it is "does the Bicep, the repo, the logs, or the write-up contain a secret value?" This PoC passes that test: the Bicep contains only references to secrets, not values.

5.4 NSG deny-all default with effective-rule verification

Design. The internal NSG cpcs-nsg-internal has a deny-all default (p4096) with two scoped allow rules:

Verification. az network nsg rule list + az network vnet subnet show confirmed all three rules are declared and the NSG is attached to internal-ingress (the subnet the ILB frontend lives on). The nodepool NSG was verified to have 0 rules — the nodes themselves have no extra inbound restrictions from an NSG; their security comes from the VNet topology (no public ingress path to the nodes, only outbound via the SNAT IP).

Finding (not a gap, but documented). The aks-node subnet has no NSG attached. This is by design — the nodes only need outbound connectivity (via the LB SNAT) and the internal-ingress NSG gates external → ILB traffic. However, a reader should not assume the deny-all p4096 rule also covers the aks-node subnet; it does not.

az network nic list-effective-nsg could not be run because AKS VNet-integrated VMSS nodes do not expose their NICs as standalone resources. The effective-rule check is therefore based on: (a) declared rules on the attached NSG, (b) the empty nodepool NSG, and (c) the VNet topology (no public ingress path to nodes). This is the maximum available without SSH access to a node.

5.5 Budget-cap discipline

Design. A $300 / month Azure Budget with alerts at 50 / 80 / 100 % to the budget-alert recipient.

Reality. The budget cannot be created via ARM for this tenant (§4.3). It is created manually in the Portal after deploy. The budget object still exists at subscription scope after teardown (see §10).

Discipline demonstrated. The $300 / mo cap was a hard constraint on the design: the 3-day PoC estimate (~$40–55 with the gateway; ~$25–35 after the gateway was dropped) was comfortably under the cap. If the estimate had exceeded the cap, the design would have been revised before deploy — not after.

6. What was intentionally out of scope

This is a scope boundary, not an oversight. The infrastructure + identity plumbing was provisioned and verified. A working end-to-end login flow would be a follow-on piece of work, not a missing requirement of this PoC.

7. Cost reconciliation

Status (as of the 2026-09-06 publish finalization): finalized as a retail-rate estimate — see §7.3 for the final cost basis and §7.4 for the closed cost-actuals note.

7.1 Evidence

Why actuals are not settled. Azure cost reporting has a 24–48 h lag for new resources. The PoC was live ~1.4 h at the verification-pass time (2026-08-26) and was torn down the same day. As of the 2026-08-27 18:10 CDT re-check (~44 h after teardown), the usage-row metadata has landed (see table above) but the prices have not — every pretaxCost / usageQuantity is still "None" — and the Cost Management query API remains unreachable from this environment (429 client-type quota / 400 schema validation), so no cost figure can be extracted.

7.2 Estimate (the number in this write-up)

• 3-day PoC estimate: ~$25–35 all-in (post-gateway-drop).
• Published figure (approved 2026-08-26, approval 4): ~$40–55 for the PoC window — the original full-stack estimate that included the VPN gateway line (~$142 / mo). After the gateway was dropped (decision b, 2026-08-25), the project's own recompute for the same 3-day window is ~$25–35 — see PROJECT.md §4. The published number follows the approved figure; both are well under the $300 / mo budget cap.

Note (attached to the estimate in the published write-up, per approval 4): This is a retail-rate estimate (Azure Retail Prices API, East US, retail USD). The actual billed cost for the PoC window can be pulled from Azure Cost Management after the fact — cost data remains queryable after resource deletion — and will be appended as a dated addendum (see §7.4 follow-up). This estimate does not block publishing.

7.3 Final cost basis

Final cost basis (as of the 2026-09-06 publish finalization): a retail-rate estimate of ~$25–35 for the ~3-day PoC window (post-gateway-drop). The PoC environment was torn down shortly after validation, and actual billed cost was not confirmed before publish. This figure is a retail-rate estimate, not a confirmed actual — treat it as directionally accurate rather than exact. It sits comfortably under the $300 / month budget cap. No actual-cost figure has been fabricated.

7.4 Cost-actuals follow-up (closed)

Closed — no open follow-up. Actual billed cost was not confirmed before publish, so the estimate in §7.3 stands as the final published figure. Cost data remains queryable in Azure Cost Management after the fact; if a confirmed figure is obtained later, it may be appended here as a dated addendum. No pending re-check remains.

8. Teardown & cleanup

8.1 AKS PoC teardown (executed 2026-08-26, approval 5)

Executed in the approved order. Every step verified against the live API, not by assumption. Raw logs: ops-teardown-verify.log, ops-teardown-revoke.log, ops-teardown-abac.log, ops-teardown-kv.log.

8.1.1 Pre-flight — no background jobs / open sessions against cpcs-rg

$ ps aux | grep -iE "az |azurerm|kubectl|kubelogin|deploy.sh|enable-insights"
(no matches — exit 1)
$ process list   # Hermes background sessions
→ only two old deploy.sh sessions, both status: exited (exit 0/1, Aug 25)

No live az, kubectl, or deploy processes. Only active CLI context is the <deployment-service-principal> SP login used for the API calls themselves.

8.1.2 Resource group deletion

$ az group delete -g cpcs-rg --yes --no-wait        # accepted, async
$ az group show -g cpcs-rg                          # in progress: "Deleting"
$ az group show -g MC_cpcs-rg_cpcs-aks_eastus       # → ResourceGroupNotFound (node RG gone first)
$ az group show -g cpcs-rg                          # → ResourceGroupNotFound (final)
ERROR: (ResourceGroupNotFound) Resource group 'cpcs-rg' could not be found.

VERIFIED DELETED — both the user RG and the AKS-managed node RG return not-found. Not merely "delete accepted."

8.1.3 Key Vault soft-delete (expected behavior, not an error)

• az keyvault list → [] — no active cpcs* vaults remain.
• Subscription-wide residual check (verified 2026-08-26): az resource list filtered on name contains cpcs → 0 rows — the only surviving artifact is the soft-deleted Key Vault <kv-name> (soft-delete 90 days, purge protection ON). Key Vault soft-delete survives RG deletion and is not visible to <deployment-service-principal> (no purge rights — the deleted-list endpoints require higher entitlement and returned 404 / 403 for this SP). This is expected: the soft-deleted vault self-resolves after the 90-day retention window. Documented as expected behavior, not an anomaly.

8.1.4 Role revocation — <deployment-service-principal> (subscription scope)

Pre-revocation live check (assignment list, SP object ID <deployment-service-principal-object-id>, subscription <SUBSCRIPTION_ID>):

Manual step (<org-owner> — performed by the human Owner, not by <deployment-service-principal> itself):

az role assignment delete   --assignee <deployment-service-principal-object-id>   --role "User Access Administrator"   --scope /subscriptions/<SUBSCRIPTION_ID>

Run as any subscription Owner / Administrator (e.g. the Microsoft account behind <org-admin-email>), then re-run az role assignment list --assignee <deployment-service-principal-object-id> to confirm zero rows. Rationale it's safe: the SP has no remaining resource-group scope after teardown; the only live entitlement is this one subscription-level assignment. (Alternatively disable the SP: az ad sp update --id <deployment-service-principal-client-id> --set disabled=true — immediate, reversible.)

Status (verified 2026-08-27): the UAM assignment has been deleted and independently re-verified — a fresh az role assignment list for this SP (subscription scope) returns empty, so <deployment-service-principal> holds zero standing role assignments. This closes the last item from the AKS PoC teardown. No open items remain from the AKS PoC teardown.

8.2 Post-outage cleanup (the AD / DC environment, 2026-08-26)

This is a separate environment (on-prem AD / DC, not the AKS PoC), but it is part of the same engagement's cleanup scope. The power outage on 2026-08-26 left the following residue:

8.2.1 Dead PoC neighbor removal (<poc-vm>)

The Azure-DC PoC VM (<poc-vm>, <poc-vm-ip>) had been torn down the previous night, but its AD metadata (connection objects, DSA tree, DNS records, computer / server / DFSR objects) was still present on both DCs and was the source of the error 1722 / error 58 replication failures.

Removed (both DCs, exact-DN only, live DCs guarded intact):

After the removal, the forest-wide replsummary was clean: DC1 49m:35s 0/5 0; DC2 42m:31s 0/5 0. The 58 - <poc-vm> line in replsummary's "operational errors" section disappeared.

8.2.2 strongSwan residue removal

The Azure-DC PoC had left a conn azure-dc-poc block in /etc/ipsec.conf (lines 26–41) and an Azure PSK line in /etc/ipsec.secrets. Removed with a backup gate:

• Backup gate PASS: /etc/ipsec.conf.bak.<pre-cleanup> present (1601 bytes, 600 perms); fresh backup /etc/ipsec.conf.bak.<post-cleanup> created.
• /etc/ipsec.conf: conn azure-dc-poc block removed. Post-change grep -c "azure-dc-poc" → 0.
• /etc/ipsec.secrets: Azure PSK line removed. Post-change grep -c "azure" → 0.
• ipsec status: Security Associations (0 up, 0 connecting): none. strongswan-starter active.

8.2.3 Stale health-check disable

dc2-reboot-verify systemd user service: stopped, disabled, reset-failed. Final state: inactive / disabled.

8.2.4 KVM autostart fix

Both DC VMs (DC1-VM = DC1, DC2-VM = DC2) verified: State: running + Autostart: enable. The outage exposed that the DCs were not set to autostart on the libvirt host — they had to be started manually after the host reboot. Fixed by enabling autostart on both VMs.

Lesson. Enable KVM autostart on any VM providing production services, not just test VMs. A domain controller that does not come back after a host reboot is a single point of failure for the entire AD forest.

8.3 Teardown summary

9. Lessons for future PoCs

1. Provision subnets serially or with explicit dependsOn from the start. Azure's Network service rejects concurrent writes to subnets of the same VNet with AnotherOperationInProgress / Conflict. Do not rely on Bicep's implicit dependency ordering for resources that the underlying service rejects in parallel. (§4.1)
2. Check regional SKU availability before designing around a specific gateway or SQL SKU. Query az aks get-versions / az sql servers list-skus / az network vpn-gateway list-skus for the target region and subscription before committing a design. East US rejected both the VPN Gateway (VpnGw1 / VpnGw2 Standard) and new SQL provisioning for this subscription. (§4.2, §4.4)
3. Verify Microsoft.Resources/budgets is exposed in the target tenant before relying on budget-as-code. It is not exposed for this tenant (404 at sub + tenant scope). Have a Portal fallback. (§4.3)
4. Enable KVM autostart on any VM providing production services, not just test VMs. A domain controller that does not come back after a host reboot is a single point of failure for the entire AD forest. (§8.2.4)
5. AAD RBAC grants on AKS can lag the API server by 15+ minutes. If you're verifying cluster access right after a grant, either wait longer or verify at the AAD layer (role assignment) rather than the API-server layer. (§4.5)
6. The API can return HTTP 200 on a PATCH that does nothing. Always re-read the field you tried to set. Container Insights binding returned 200 but addonProfiles stayed null. (§4.5)
7. A service principal with UAM cannot revoke its own UAM grant. The ABAC condition on the assignment forbids it. Plan for an external Owner / Admin to perform the final revocation, or disable the SP as an immediate stopgap. (§5.2, §8.1.4)
8. A frozen replication state on one DC while the other is green is usually a client-side / transport problem, not directory corruption. Verify the services, clear the orphans, run a bounded scoped sync, and only escalate to metadata surgery if the scoped sync still shows failures on a live neighbor. Do not reach for USN rollback or authoritative restore first. (§4.6)
9. Cost data has a 24–48 h lag. Do not block publishing on actuals — use a retail-rate estimate with an explicit note, and pull actuals as a dated addendum once the lag clears. (§7)
10. The test of a no-secrets design is not "is there a Key Vault?" — it is "does the Bicep, the repo, the logs, or the write-up contain a secret value?" (§5.3)
11. Before relying on a terminal's Azure CLI session for privileged operations (especially role-assignment changes), explicitly confirm the active identity via az account show — a session can silently be authenticated as a service principal rather than the intended human account, which is exactly what caused §10.8 to appear unresolvable until the identity mismatch was found. (§10.8)

10. Inconsistencies & open items flagged during assembly

These are items the writer flagged rather than silently resolved, per the task's instruction to "flag anything in the source material that seemed inconsistent or unclear."

10.1 Deployment-start timestamp discrepancy

• PROJECT.md §10.5 states: "deploy 2026-08-25T20:24Z → teardown 2026-08-26."
• ops-teardown-abac.log (the ABAC condition createdOn field on the UAM role assignment) states: "2026-08-25T22:33:00.261895+00:00."
• verification-report.md Check 5 states: "Deployment start 2026-08-26T01:50:44Z" (this is later than both of the above, which is inconsistent — the verification pass was run on 2026-08-26, so the deployment must have started before that; the 01:50:44Z figure appears to be a different event, possibly the verification pass itself being mislabeled as the "deployment start").

Resolution for this write-up. I used 2026-08-25T22:33Z (the createdOn timestamp on the UAM role assignment, which is the most authoritative single timestamp in the source material — it is the moment the role was granted, which must have been immediately before or during the deploy). The 20:24Z figure from PROJECT.md is 2 h 9 min earlier and may reflect an earlier failed deploy attempt. This is flagged, not silently resolved.

10.2 Cost actuals — finalized as a labeled estimate (closed 2026-09-06)

Actual billed cost was not confirmed before publish. The write-up therefore uses a retail-rate estimate (~$25–35 for the ~3-day PoC window, post-gateway-drop) as the final published figure, clearly labeled as an estimate and not a confirmed actual (see §7.3). The original ~$40–55 figure included the VPN gateway line that was dropped on 2026-08-25. This is a labeled estimate, not a guess, and no actual-cost figure is fabricated. The cost-actuals follow-up is closed (see §7.4); cost data remains queryable in Azure Cost Management after the fact should a confirmed figure ever be obtained.

10.3 "Resource-visualizer" architecture diagrams do not exist as files

The task instruction was to "include the two architecture diagrams already generated (embed / reference the resource-visualizer images)." No such images exist in the AKS project folder. The only azure-resource-map.png / architecture-diagram.svg files in this box belong to the Domain Controller case study (a different project, in ~/Downloads/). The azure-resource-visualizer skill exists in the Cursor plugin cache but was not used to generate any AKS diagrams in this project.

Resolution for this write-up. I generated the two diagrams directly from the authoritative Bicep source (main.bicep + cluster.bicep) as Mermaid code blocks (Figure 1 and Figure 2 in §2.1). They are accurate to the source but are newly generated, not "already generated." This is flagged, not fabricated. If the original "resource-visualizer" images exist elsewhere (a different machine, a different folder, a different project), they should be located and substituted.

10.4 Task instruction says "2 × D2s_v5" but the source says D2s_v7

The task instruction states: "AKS (2 × D2s_v5, single namespace)." The actual deployed and verified VM size is Standard_D2s_v7 (see cluster.bicep line 48, main.bicep line 48, and verification-report.md Check 4: "1 pool: node, 2 × Standard_D2s_v7"). D2s_v5 was not allowed in East US for this subscription (preflight VM size not allowed error, §4.4). This write-up uses D2s_v7 (the real value). This is flagged, not silently "corrected" to match the task instruction.

10.5 Task instruction says "single namespace" but the source does not mention namespaces

The task instruction states: "AKS (2 × D2s_v5, single namespace)." The Bicep source and the verification report do not mention a single namespace. The cluster has one node pool (node, 2 nodes) but the source does not state how many namespaces exist (AKS defaults to default, kube-system, kube-public, kube-node-lease, and any AKS-managed namespaces). This write-up describes the cluster as "single node pool" (the verifiable fact) and does not claim "single namespace." This is flagged, not assumed.

10.6 Task instruction says "single database with schemas" but the source does not mention schemas

The task instruction states: "SQL Serverless (centralus, single DB with schemas)." The Bicep source and the verification report confirm a single database (employeedb) but do not mention schemas. The portal application (out of scope for this PoC) would define the schemas. This write-up describes it as "single database employeedb" (the verifiable fact) and does not claim "with schemas." This is flagged, not assumed.

10.7 SQL Serverless v1 vs v2

• cluster.bicep line 365: "SQL (SERVERLESS, General Purpose)" — the comment says v1 (line 366: "At @2021-11-01, Serverless v1 requires: …").
• Checkpoint note, line 101: "SQL centralus Serverless v2 (auto-pause 3600s)."

Resolution for this write-up. I used Serverless v1 (the Bicep source is authoritative; the checkpoint is a narrative fragment). This is flagged.

10.8 UAM residual step — RESOLVED (manual, by design) — verified 2026-08-27

The User Access Administrator role assignment (<role-assignment-uam>) on <deployment-service-principal> has been deleted and independently re-verified. As of 2026-08-27, az role assignment list --scope /subscriptions/<SUBSCRIPTION_ID> --assignee <deployment-service-principal-object-id> returns [] — <deployment-service-principal> holds zero standing role assignments at subscription scope. (The remaining subscription-scope rows are all human Owner assignments — one on the owner MSA plus one on the owner work account. A duplicate Owner row on the owner MSA, created 135 ms after the original by a double-executed grant, was also identified and deleted, leaving exactly two Owner rows.)

Root cause (confirmed, not assumed): the CLI session used for the original revocation attempt was itself authenticated as the <deployment-service-principal> service principal (client ID <deployment-service-principal-client-id>, app-only token), not as a human account. Two deliberate locks made self-resolution impossible for that SP:

1. Permission gap — Microsoft.Authorization/roleAssignments/delete is not in the User Access Administrator role definition; UAM grants read + create, not delete. Delete requires Owner.
2. ABAC self-protection — the assignment additionally carries the by-design GuidNotEquals {<role-def-uam-condition-1>, <role-def-uam-condition-2>, <role-def-uam-condition-3>} condition on write/delete, explicitly excluding its own holder. A deliberate double-lock, not a bug.

Failed autonomous attempt, 2026-08-26 (for continuity with the incident narrative):

HTTP 403 AuthorizationFailed — "The client '<deployment-service-principal-client-id>' with object id '<deployment-service-principal-object-id>' has an authorization with ABAC condition that is not fulfilled to perform action 'Microsoft.Authorization/roleAssignments/delete' over scope …"

Resolution (manual, by the user, under their own Owner account): after identifying that the terminal's active Azure CLI session was unexpectedly authenticated as the SP rather than the user — az account show confirmed the SP was active; an interactive az login switched the session to the human Owner account (<owner-msa-email>) — the delete was then run successfully:

az role assignment delete --ids /subscriptions/<SUBSCRIPTION_ID>/providers/Microsoft.Authorization/roleAssignments/<role-assignment-uam>

Verified via az role assignment list for the assignee returning an empty result ([]) — re-run independently by the automation on 2026-08-27 (which was then operating under the human Owner session) with the same empty result.

Note: this was correctly not resolved autonomously by the Hermes agent, per the standing constraint against creating new privileged identities or working around the block — the fix required a genuinely privileged human session, which the automation does not have access to in this environment. This is the intended, safe outcome for this class of exposure, not a shortfall. It closes the last open item from the AKS PoC teardown, and it validates the §8.1.4 anticipation: a SP cannot revoke its own UAM; the final revocation is by design performed by an external human Owner.

10.9 Azure Budget object still present

The $300 / month Azure Budget (created manually in the Portal, since ARM 404 for this tenant) is still present at subscription scope after teardown. The PoC cost lines are moot post-teardown, but the budget object itself has not been deleted. This is flagged. If the budget is no longer needed, it should be deleted in the Portal.

— End of case study —