AKS Update to 1.26.0 (#374)

WAF Policy Update
Various API Updates
Linter Updates
Add a few walkthrough notes
Update Alerts and add data cap alert for LA

Author: ckittel

01-prerequisites.md

@@ -30,19 +30,16 @@ This is the starting point for the instructions on deploying the [AKS baseline r
 
 1. While the following feature(s) are still in _preview_, please enable them in your target subscription.
 
-   1. [Register the Defender for Containers preview feature = `AKS-AzureDefender`](https://learn.microsoft.com/azure/defender-for-cloud/defender-for-containers-enable?pivots=defender-for-container-aks&tabs=k8s-deploy-cli%2Ck8s-deploy-asc%2Ck8s-verify-asc%2Ck8s-remove-arc%2Caks-removeprofile-api#deploy-the-defender-profile)
-
    1. [Register the Workload Identity preview feature = `EnableWorkloadIdentityPreview`](https://learn.microsoft.com/azure/aks/workload-identity-deploy-cluster#register-the-enableworkloadidentitypreview-feature-flag)
 
-   1. Register the ImageCleaner (Earser) preview feature = `EnableImageCleanerPreview`](https://learn.microsoft.com/azure/aks/image-cleaner#prerequisites)
+   1. [Register the ImageCleaner (Earser) preview feature = `EnableImageCleanerPreview`](https://learn.microsoft.com/azure/aks/image-cleaner#prerequisites)
 
    ```bash
-   az feature register --namespace "Microsoft.ContainerService" -n "AKS-AzureDefender"
    az feature register --namespace "Microsoft.ContainerService" -n "EnableWorkloadIdentityPreview"
    az feature register --namespace "Microsoft.ContainerService" -n "EnableImageCleanerPreview"
 
    # Keep running until all say "Registered." (This may take up to 20 minutes.)
-   az feature list -o table --query "[?name=='Microsoft.ContainerService/AKS-AzureDefender' || name=='Microsoft.ContainerService/EnableWorkloadIdentityPreview' || name=='Microsoft.ContainerService/EnableImageCleanerPreview'].{Name:name,State:properties.state}"
+   az feature list -o table --query "[?name=='Microsoft.ContainerService/EnableWorkloadIdentityPreview' || name=='Microsoft.ContainerService/EnableImageCleanerPreview'].{Name:name,State:properties.state}"
 
    # When all say "Registered" then re-register the AKS resource provider
    az provider register --namespace Microsoft.ContainerService

03-aad.md

@@ -29,7 +29,7 @@ This does not configure anything related to workload identity. This configuratio
 
 1. Playing the role as the Contoso Bicycle Azure AD team, login into the tenant where Kubernetes Cluster API authorization will be associated with.
 
-   > :bulb: Skip the `az login` command if you plan to use your current user account's Azure AD tenant for Kubernetes authorization.
+   > :bulb: Skip the `az login` command if you plan to use your current user account's Azure AD tenant for Kubernetes authorization. _Using the same tenant is common._
 
    ```bash
    az login -t <Replace-With-ClusterApi-AzureAD-TenantId> --allow-no-subscriptions

04-networking.md

@@ -104,6 +104,12 @@ The following two resource groups will be created and populated with networking
    >
    > Hubs and spokes are controlled by the networking team's GitHub Actions workflows. This automation is not included in this reference implementation as this body of work is focused on the AKS baseline and not the networking team's CI/CD practices.
 
+## Private DNS Zones
+
+Private DNS zones in this reference implementation are implemented directly at the spoke level, meaning the workload team creates the private link DNS zones & records for the resources needed; furthermore, the workload is directly using Azure DNS for resolution. Your networking topology might support this decentralized model or instead DNS & DNS zones for Private Link might be handed at the regional hub or in a [VWAN virtual hub extension](https://learn.microsoft.com/azure/architecture/guide/networking/private-link-vwan-dns-virtual-hub-extension-pattern) by your networking team.
+
+If your organization operate a centeralized DNS model, you'll need to adapt how DNS zones records are managed this implementation into your existing enteprise networking DNS zone strategy. Since this reference implementation is expected to be deployed isolated from existing infrastructure; this is not something you need to address now; but will be something to understand and address when taking your solution to production.
+
 ### Next step
 
 :arrow_forward: [Prep for cluster bootstrapping](./05-bootstrap-prep.md)

05-bootstrap-prep.md

@@ -12,10 +12,13 @@ Container registries often have a lifecycle that extends beyond the scope of a s
 
 The role of this pre-existing ACR instance is made more prominant when we think about cluster bootstrapping. That is the process that happens after Azure resource deployment of the cluster, but before your first workload lands in the cluster. The cluster will be bootstrapped _immedately and automatically_ after resource deployment, which means you'll need ACR in place to act as your official OCI artifact repository for required images and Helm charts used in that bootstrapping process.
 
-### Method
+### Bootstrapping method
 
 We'll be bootstrapping this cluster with the Flux GitOps agent as installed as an AKS extension. This specific choice does not imply that Flux, or GitOps in general, is the only approach to bootstrapping. Consider your organizational familiarity and acceptance of tooling like this and decide if cluster bootstrapping should be performed with GitOps or via your deployment pipelines. If you are running a fleet of clusters, a GitOps approach is highly recommended for uniformity and easier governance. When running only a few clusters, GitOps might be seen as "too much" and you might instead opt for integrating that process into one or more deployment pipelines to ensure bootstrapping takes place. No matter which way you go, you'll need your bootstrapping artifacts ready to go before you start your cluster deployment so that you can minimize the time between cluster deployment and bootstrapping. Using the Flux AKS extension allows your cluster to start already bootstrapped and sets you up with a solid management foundation going forward.
 
+### Additional resources
+
+In addition to ACR being deployed to support bootstrapping, this is where any other resources that are considered not tied to the lifecycle of an individual cluster is deployed. ACR is one example as talked about above. Another example could be an AKS Backup Vault and backup artifacts storage account which likely would exist prior to and after any individual AKS cluster's existance. When designing your pipelines, ensure to isolate components by their lifecycle watch for singletons in an architecture. These are typically resources like regional logging sinks, supporting global routing infrastructure, etc. This is in contrast to potentially transiently/replaceable components, like the AKS cluster itself. _This implemention does not represent a complete seperation of stamp vs regional resources, but is fairly close. Deviations are strickly for ease of deployment in this walkthrough instead of as examples of guidance._
 ## Steps
 
 1. Create the AKS cluster resource group.
@@ -38,7 +41,7 @@ We'll be bootstrapping this cluster with the Flux GitOps agent as installed as a
    echo RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE: $RESOURCEID_VNET_CLUSTERSPOKE_AKS_BASELINE
    ```
 
-1. Deploy the container registry template.
+1. Deploy the container registry and non-stamp resources template.
 
    ```bash
    # [This takes about four minutes.]

acr-stamp.bicep

@@ -58,37 +58,78 @@ var subRgUniqueString = uniqueString('aks', subscription().subscriptionId, resou
 
 /*** EXISTING RESOURCES ***/
 
-resource spokeResourceGroup 'Microsoft.Resources/resourceGroups@2021-04-01' existing = {
+resource spokeResourceGroup 'Microsoft.Resources/resourceGroups@2022-09-01' existing = {
   scope: subscription()
-  name: '${split(targetVnetResourceId,'/')[4]}'
+  name: split(targetVnetResourceId,'/')[4]
 }
 
-resource spokeVirtualNetwork 'Microsoft.Network/virtualNetworks@2021-05-01' existing = {
+resource spokeVirtualNetwork 'Microsoft.Network/virtualNetworks@2022-09-01' existing = {
   scope: spokeResourceGroup
-  name: '${last(split(targetVnetResourceId,'/'))}'
+  name: last(split(targetVnetResourceId,'/'))
 
-  resource snetPrivateLinkEndpoints 'subnets@2021-05-01' existing = {
+  resource snetPrivateLinkEndpoints 'subnets' existing = {
     name: 'snet-privatelinkendpoints'
   }
 }
 
 /*** RESOURCES ***/
 
 // This Log Analytics workspace will be the log sink for all resources in the cluster resource group. This includes ACR, the AKS cluster, Key Vault, etc. It also is the Container Insights log sink for the AKS cluster.
-resource laAks 'Microsoft.OperationalInsights/workspaces@2021-06-01' = {
+resource laAks 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
   name: 'la-aks-${subRgUniqueString}'
   location: location
   properties: {
     sku: {
       name: 'PerGB2018'
     }
     retentionInDays: 30
+    publicNetworkAccessForIngestion: 'Enabled'
+    publicNetworkAccessForQuery: 'Enabled'
+    workspaceCapping: {
+      dailyQuotaGb: -1 // No daily cap (configure alert below if enabled)
+    }
+  }
+}
+
+// Add a alert rule if the log analytics workspace daily data cap has been reached.
+// Logging costs can be a significant part of any architecture, and putting a cap on
+// a logging sink (none of which are applied here), can help keep costs in check but
+// you run a risk of losing critical data.
+resource sqrDailyDataCapBreach 'Microsoft.Insights/scheduledQueryRules@2018-04-16' = {
+  name: 'Daily data cap breached for workspace ${laAks.name} CIQ-1'
+  location: location
+  properties: {
+    description: 'This alert monitors daily data cap defined on a workspace and fires when the daily data cap is breached.'
+    displayName: 'Daily data cap breached for workspace ${laAks.name} CIQ-1'
+    enabled: 'true'
+    source: {
+      dataSourceId: laAks.id
+      queryType: 'ResultCount'
+      authorizedResources: []
+      query: '_LogOperation | where Operation == "Data collection Status" | where Detail contains "OverQuota"'
+    }
+    schedule: {
+      frequencyInMinutes: 5
+      timeWindowInMinutes: 5
+    }
+    action: {
+      'odata.type': 'Microsoft.WindowsAzure.Management.Monitoring.Alerts.Models.Microsoft.AppInsights.Nexus.DataContracts.Resources.ScheduledQueryRules.AlertingAction'
+      severity: '1'
+      aznsAction: {
+        actionGroup: []
+      }
+      throttlingInMin: 1440
+      trigger: {
+        threshold: 0
+        thresholdOperator: 'GreaterThan'
+      }
+    }
   }
 }
 
 // Apply the built-in 'Container registries should have anonymous authentication disabled' policy. Azure RBAC only is allowed.
 var pdAnonymousContainerRegistryAccessDisallowedId = tenantResourceId('Microsoft.Authorization/policyDefinitions', '9f2dea28-e834-476c-99c5-3507b4728395')
-resource paAnonymousContainerRegistryAccessDisallowed 'Microsoft.Authorization/policyAssignments@2021-06-01' = {
+resource paAnonymousContainerRegistryAccessDisallowed 'Microsoft.Authorization/policyAssignments@2022-06-01' = {
   name: guid(resourceGroup().id, pdAnonymousContainerRegistryAccessDisallowedId)
   location: 'global'
   scope: resourceGroup()
@@ -107,7 +148,7 @@ resource paAnonymousContainerRegistryAccessDisallowed 'Microsoft.Authorization/p
 
 // Apply the built-in 'Container registries should have local admin account disabled' policy. Azure RBAC only is allowed.
 var pdAdminAccountContainerRegistryAccessDisallowedId = tenantResourceId('Microsoft.Authorization/policyDefinitions', 'dc921057-6b28-4fbe-9b83-f7bec05db6c2')
-resource paAdminAccountContainerRegistryAccessDisallowed 'Microsoft.Authorization/policyAssignments@2021-06-01' = {
+resource paAdminAccountContainerRegistryAccessDisallowed 'Microsoft.Authorization/policyAssignments@2022-06-01' = {
   name: guid(resourceGroup().id, pdAdminAccountContainerRegistryAccessDisallowedId)
   location: 'global'
   scope: resourceGroup()
@@ -130,12 +171,12 @@ resource dnsPrivateZoneAcr 'Microsoft.Network/privateDnsZones@2020-06-01' = {
   location: 'global'
   properties: {}
 
-  resource dnsVnetLinkAcrToSpoke 'virtualNetworkLinks@2020-06-01' = {
+  resource dnsVnetLinkAcrToSpoke 'virtualNetworkLinks' = {
     name: 'to_${spokeVirtualNetwork.name}'
     location: 'global'
     properties: {
       virtualNetwork: {
-        id: targetVnetResourceId
+        id: spokeVirtualNetwork.id
       }
       registrationEnabled: false
     }
@@ -209,8 +250,8 @@ resource acrAks_diagnosticsSettings 'Microsoft.Insights/diagnosticSettings@2021-
   }
 }
 
-// Expose Azure Container Registry via Private Link, into the cluster nodes subnet.
-resource privateEndpointAcrToVnet 'Microsoft.Network/privateEndpoints@2021-05-01' = {
+// Expose Azure Container Registry via Private Link, into the cluster nodes virtual network.
+resource privateEndpointAcrToVnet 'Microsoft.Network/privateEndpoints@2022-09-01' = {
   name: 'pe-${acrAks.name}'
   location: location
   dependsOn: [
@@ -233,7 +274,7 @@ resource privateEndpointAcrToVnet 'Microsoft.Network/privateEndpoints@2021-05-01
     ]
   }
 
-  resource privateDnsZoneGroupAcr 'privateDnsZoneGroups@2021-05-01' = {
+  resource privateDnsZoneGroupAcr 'privateDnsZoneGroups' = {
     name: 'default'
     properties: {
       privateDnsZoneConfigs: [