10.7 Full-Domain Case Lab

Scenario brief

Contoso has a production subscription under a management group. The company runs a web application on App Service, a background processor in Azure Container Apps, two administration VMs without public IP addresses, a storage account for reports, and a hub-spoke network. Security requires least privilege, no public storage access, no public VM management ports, and repeatable deployments through Bicep. Operations requires alerts for failed backups, unhealthy load-balanced services, and storage availability. The business asks you to prepare a remediation runbook for a recent outage.

Recent changes were made by different teams. A policy assignment now denies storage accounts with public network access enabled. A CanNotDelete lock was placed on the production resource group. The storage account now uses a private endpoint for blob and file. A Container App revision was deployed with a new image tag. App Service was switched from storage account keys to managed identity. Administrators are expected to use Azure Bastion for VM access.

Current symptoms:

1. The App Service app returns 403 when reading report blobs.
2. The Container App revision does not start because it cannot pull the image.
3. The report processor also cannot mount the Azure Files share.
4. Administrators cannot connect to the VMs through Bastion.
5. A Bicep deployment fails when creating a role assignment.
6. Backup failures were not noticed for two days.

Step 1: Classify each symptom

Do not solve them in the order people complain. Classify by failed operation.

This table prevents shotgun remediation. A 403 is not the same as a timeout. A deployment authorization error is not fixed by changing the workload identity. A backup monitoring gap is not fixed by restoring a VM.

Step 2: Remediate identity and storage access

For App Service, confirm the system-assigned or user-assigned managed identity used by the app. Then check role assignments on the storage account or container. If the app only reads report blobs, assign Storage Blob Data Reader at the container scope. If it writes reports, use Storage Blob Data Contributor. Do not return to storage account keys because the requirement is least privilege and no secrets.

Also validate storage networking. If public access is disabled, App Service must have a supported private network path to the storage endpoint, or the architecture must use an approved access pattern. If the error is truly 403 after reaching the service, role assignment is primary. If logs show timeout or name resolution failure, private endpoint DNS and network integration become primary.

Step 3: Remediate Container App startup

For image pull, confirm the image reference: <registry>.azurecr.io/<repository>:<tag>. Confirm the tag exists in ACR. Confirm the Container App identity is assigned AcrPull at the registry scope. If ACR uses firewall or private endpoint restrictions, confirm the Container Apps environment has the required network path and DNS.

For Azure Files mount, check the storage account, file share, secret or identity configuration, and private endpoint for the file subresource. A blob private endpoint does not provide SMB file share access. Confirm DNS from the workload network resolves the file endpoint privately when public access is disabled. If using account keys in a secret, confirm the key is current; if the requirement prohibits keys, configure the supported identity-based method instead.

Step 4: Remediate Bastion administration

Azure Bastion must be deployed into a subnet named AzureBastionSubnet. Validate subnet size, Bastion resource health, public IP requirements for the Bastion service itself, and NSG rules if applied. The VMs do not need public IP addresses. Restoring public IPs to the VMs violates the security requirement.

If administrators cannot connect, check role permissions to use the VM and Bastion, VM power state, guest OS firewall, NSGs on VM subnets, and Bastion subnet rules. Use the portal path Virtual machine > Connect > Bastion only after the Bastion resource is correctly deployed.

Step 5: Fix Bicep deployment authorization

The Bicep deployment fails when creating a role assignment. The deployment identity needs permission to create role assignments at the target scope. Contributor is not enough to grant access. Use an identity with Owner or User Access Administrator at the appropriate scope for the deployment operation, then assign least-privilege roles to workloads.

Do not give the application Owner to work around deployment failure. Separate deployment authority from runtime authority. Deployment authority creates the infrastructure and role assignments. Runtime authority performs the app's limited data operations.

Step 6: Restore monitoring and maintenance

Backup failures went unnoticed because alerts were missing or misrouted. Review Recovery Services vault > Backup jobs, protected items, policy, and vault alerts. Create or fix alert rules and action groups for backup job failures. Confirm email, webhook, ITSM, or automation receivers are current. Use alert processing rules for planned maintenance, not for hiding real failures.

Add monitoring for the other outage indicators: Container App revision failures, App Service HTTP errors, storage availability and latency, private endpoint connection health where available, and load balancer backend health if load-balanced services exist. Monitoring should detect the next failure at the failing layer.

Step 7: Produce the runbook answer

A good final answer is operational:

1. App Service blob 403: assign the correct blob data role to the app managed identity at the narrowest scope and validate private storage access.
2. Container image pull: assign AcrPull to the Container App identity, verify image tag, and validate registry network path.
3. Azure Files mount: verify file private endpoint, private DNS, share configuration, and secret or identity-based authorization.
4. Bastion: deploy or repair Azure Bastion in AzureBastionSubnet and keep VM public IPs removed.
5. Bicep role assignment: run deployment with an identity permitted to create role assignments; keep workload roles least privileged.
6. Backup alerting: configure backup failure alerts and action groups, then review failed jobs and recovery points.
7. Governance: respect policy and locks; change them only through approved maintenance when they block a legitimate operation.

Exam reflection

This lab touches every AZ-104 domain. Identity decides who can grant access and what workloads can do. Storage supplies data services and network restrictions. Compute hosts the web app, containers, and VMs. Networking controls private endpoints, Bastion, DNS, and packet flow. Monitoring and maintenance detect failures, backup problems, and health changes.

The core skill is not memorizing a single service name. It is preserving constraints while fixing the right layer. If the requirement says no public storage access, do not enable public access. If the requirement says least privilege, do not assign Owner to the app. If the requirement says repeatable deployment, fix the Bicep deployment identity instead of clicking around without source control. That is the administrator behavior AZ-104 is trying to measure.