Skip to content

Commit 4a23122

Browse files
k8s-ci-robotk8s-ci-robot
authored
Merge pull request #5463 from k8s-infra-cherrypick-robot/cherry-pick-5451-to-release-1.18
[release-1.18] Tiltfile aks as mgmt improvements
2 parents fb2557c + 1c28e12 commit 4a23122

File tree

7 files changed

+474
-110
lines changed

7 files changed

+474
-110
lines changed

‎Makefile

+1
Original file line numberDiff line numberDiff line change
@@ -753,6 +753,7 @@ kind-create: $(KUBECTL) ## Create capz kind cluster if needed.
753753
.PHONY: aks-create
754754
aks-create: $(KUBECTL) ## Create aks cluster as mgmt cluster.
755755
./scripts/aks-as-mgmt.sh
756+
MANIFEST_IMG=$(CONTROLLER_IMG) MANIFEST_TAG=$(TAG) $(MAKE) set-manifest-image
756757

757758
.PHONY: tilt-up
758759
tilt-up: install-tools ## Start tilt and build kind cluster if needed.

‎Tiltfile

+259-61
Large diffs are not rendered by default.

‎docs/book/src/developers/development.md

+4-5
Original file line numberDiff line numberDiff line change
@@ -156,11 +156,8 @@ development will span both CAPZ and CAPI, then follow the [CAPI and CAPZ instruc
156156
<aside class="note warning">
157157
<h2>Warning</h2>
158158
<p>
159-
To use an internal load balancer (ILB) intra-cluster node-apiserver traffic in your workload cluster, follow the
160-
instructions in the
161-
<a href="#tilt-for-dev-using-internal-load-balancer-ilb-for-intra-cluster-node-apiserver-traffic">
162-
Tilt for Dev: Internal LB for Cluster VNet Communication
163-
</a> section.
159+
To use an internal load balancer (ILB) for intra-cluster node-apiserver traffic in your workload cluster, please refer to the
160+
<a href="./tilt-with-aks-as-mgmt-ilb.md">Tilt with AKS as Management Cluster</a> guide.
164161
</p>
165162
</aside>
166163

@@ -262,6 +259,8 @@ make delete-workload-cluster
262259
263260
#### Tilt for dev using internal load balancer (ILB) for intra-cluster node-apiserver traffic
264261

262+
263+
265264
This flow is for developers who want to leverage the internal load balancer for intra-cluster node-apiserver traffic.
266265
You can achieve this by setting the `EXP_APISERVER_ILB` environment variable to `true` in your shell (run `export EXP_APISERVER_ILB=true`) and then create the CAPZ management cluster.
267266

‎docs/book/src/developers/tilt-with-aks-as-mgmt-ilb.md

+124
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Tilt with AKS as Management Cluster with Internal Load Balancer
2+
3+
## Introduction
4+
This guide is explaining how to set up and use Azure Kubernetes Service (AKS) as a management cluster for Cluster API Provider Azure (CAPZ) development using Tilt and an internal load balancer (ILB).
5+
6+
While the default Tilt setup recommends using a KIND cluster as the management cluster for faster development and experimentation, this guide demonstrates using AKS as an alternative management cluster. We also cover additional steps for working with stricter network policies - particularly useful for organizations that need to maintain all cluster communications within their Azure Virtual Network (VNet) infrastructure with enhanced access controls.
7+
8+
### Who is this for?
9+
- Developers who want to use AKS as the management cluster for CAPZ development.
10+
- Developers working in environments with strict network security requirements.
11+
- Teams that need to keep all Kubernetes API traffic within Azure VNet
12+
13+
14+
**Note:** This is not a production ready solution and should not be used in a production environment. This is only meant to be used for development/testing purposes.
15+
16+
### Prerequisites
17+
- All [general CAPZ prerequisites](../getting-started.md#prerequisites) should be satisfied
18+
- Basic understanding of Azure networking concepts.
19+
- Familiarity with Cluster API and CAPZ.
20+
- Tilt installed on your development machine.
21+
- `export REGISTRY=<registry_of_your-choice>` set to the registry of your choice.
22+
- If `tilt-settings.yaml` file exists in the root of your repo, clear out any values in `kustomize_settings` unless you want to use them instead of the values that will be set by running `make aks-create`.
23+
24+
## Using Tilt with AKS as the Management Cluster
25+
26+
To use Tilt with AKS as the management cluster, you need to run the following commands:
27+
- `make clean`
28+
- `make generate`
29+
- `make acr-login`
30+
- `make aks-create`
31+
- `make tilt-up`
32+
33+
Using the tilt UI, click on the flavors you want to deploy and CAPZ will deploy the workload cluster with the selected flavor.
34+
35+
## Leveraging internal load balancer
36+
37+
By default using Tilt with Cluster API Provider Azure (CAPZ), the management cluster is exposed via a public endpoint. This works well for many development scenarios but presents challenges in environments with strict network security requirements.
38+
39+
40+
### Challenges and Solutions
41+
42+
1. **Management to Workload Cluster Connectivity**
43+
44+
**Scenario:**
45+
- Management cluster cannot connect to workload cluster's API server via workload cluster's FQDN.
46+
47+
**Solution:**
48+
- Peer management and workload cluster VNets.
49+
- Set Workload cluster API server replica count to 3. (Default is 1 when using KIND as the management cluster).
50+
- This is done by setting `CONTROL_PLANE_MACHINE_COUNT` to 3 in the tilt-settings.yaml file.
51+
- `make aks-create` will set this value to 3 for you.
52+
- Create a internal load balancer (ILB) with the workload cluster's apiserver VMs as its backend pool in the workload cluster's VNet.
53+
- This is achieved by setting `EXP_INTERNAL_LB=true`. `EXP_INTERNAL_LB` is set to `true` by default when running `make tilt-up`.
54+
- Create private DNS zone with workload cluster's apiserver FQDN as its record to route management cluster calls to workload cluster's API server private IP.
55+
- As of current release, a private DNS zone is automatically created in the tilt workflow for `apiserver-ilb` and `windows-apiserver-ilb` flavors.
56+
57+
2. **Workload Cluster Node Communication**
58+
59+
**Scenario:**
60+
- Workload cluster worker nodes should not be able to communicate with their workload cluster's API server's FQDN.
61+
62+
**Solution:**
63+
- Update `/etc/hosts` on worker nodes via preKubeadmCommands in the `KubeadmConfigTemplate` with `- echo '${AZURE_INTERNAL_LB_PRIVATE_IP} ${CLUSTER_NAME}-${APISERVER_LB_DNS_SUFFIX}.${AZURE_LOCATION}.cloudapp.azure.com' >> /etc/hosts`
64+
- This essentially creates a static route (using the ILB's private IP) to the workload cluster's API server (FQDN) in the worker nodes.
65+
- Deploying `apiserver-ilb` or `windows-apiserver-ilb` flavor will deploy worker nodes of the workload cluster with the private IP of the ILB as their default route.
66+
67+
3. **Network Security Restrictions**
68+
69+
**Scenario:**
70+
- Critical ports needed for management cluster to communicate with workload cluster.
71+
- ports:
72+
- `TCP: 443, 6443`
73+
- `UDP: 53`
74+
75+
**Solution:**
76+
- Once tilt UI is up and running, click on the `allow required ports on mgmt cluster` task to allow the required ports on the management cluster's API server.
77+
78+
4. **Load Balancer Hairpin Routing**
79+
80+
**Challenge:**
81+
- Workload cluster's control plane fails to bootstrap due to internal LB connectivity timeouts.
82+
- Single control plane node setup causes hairpin routing issues.
83+
84+
**Solution:**
85+
- Use 3 control plane nodes in a stacked etcd setup.
86+
- Using aks as management cluster sets `CONTROL_PLANE_MACHINE_COUNT` to 3 by default.
87+
88+
### Flavors leveraging internal load balancer
89+
- `cluster-template-apiserver-ilb.yaml`
90+
- `cluster-template-windows-apiserver-ilb.yaml`
91+
92+
#### Tilt Workflow for AKS as Management Cluster with Internal Load Balancer
93+
94+
- In tilt-settings.yaml, set subscription_type to "corporate" and remove any other env values unless you want to override env variables created by `make aks-create`. Example:
95+
```
96+
.
97+
.
98+
kustomize_substitutions:
99+
"SUBSCRIPTION_TYPE": "corporate"
100+
.
101+
```
102+
- `make clean`
103+
- This make target does not need to be run every time. Run it to remove bin and kubeconfigs.
104+
- `make generate`
105+
- This make target does not need to be run every time. Run it to update your go related targets, manifests, flavors, e2e-templates and addons.
106+
- Run it periodically upon updating your branch or if you want to make changes in your templates.
107+
- `make acr-login`
108+
- Run this make target if you have `REGISTRY` set to an Azure Container Registry.
109+
- `make aks-create`
110+
- Run this target to bring up an AKS cluster.
111+
- Once the AKS cluster is created, you can reuse the cluster as many times as you like. Tilt ensures that the new image gets deployed every time there are changes in the Tiltfile and/or dependent files.
112+
- Running `make aks-create` cleans up any existing `aks_as_mgmt_settings`.
113+
- `make tilt-up`
114+
- Run this target to use underlying cluster being pointed by your `KUBECONFIG`.
115+
116+
Once the tilt UI is up and running
117+
- Click on the `allow required ports on mgmt cluster` task to allow the required ports on the management cluster's API server.
118+
- Note: This task will wait for the NSG rules to be created and then update them to allow the required ports.
119+
- This task will take a few minutes to complete.
120+
- This target can run in parallel and independent of deploying any cluster flavors.
121+
- Click on the flavors you want to deploy and CAPZ will deploy the workload cluster with the selected flavor.
122+
- Flavors that leverage internal load balancer are:
123+
- `apiserver-ilb`
124+
- `windows-apiserver-ilb`

‎hack/verify-starlark.sh

+17-6
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,29 @@ set -o pipefail
2020
SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")"
2121
ROOT_PATH="$(cd "${SCRIPT_DIR}"/.. && pwd)"
2222

23-
VERSION="0.29.0"
23+
VERSION="v8.0.3"
2424

2525
MODE="check"
2626

2727
if [[ "$*" == "fix" ]]; then
2828
MODE="fix"
2929
fi
3030

31-
if [[ "${OSTYPE}" == "linux"* ]]; then
32-
BINARY="buildifier"
33-
elif [[ "${OSTYPE}" == "darwin"* ]]; then
34-
BINARY="buildifier.mac"
31+
OS="$(uname -s)"
32+
ARCH="$(uname -m)"
33+
34+
# Determine OS-specific binary name
35+
if [[ "${OS}" == "Linux" ]]; then
36+
BINARY="buildifier-linux"
37+
elif [[ "${OS}" == "Darwin" ]]; then
38+
BINARY="buildifier-darwin"
39+
fi
40+
41+
# Append architecture suffix for the appropriate binary
42+
if [[ "${ARCH}" == "x86_64" ]] || [[ "${ARCH}" == "amd64" ]]; then
43+
BINARY="${BINARY}-amd64" # No change needed, this is the default
44+
elif [[ "${ARCH}" == "arm64" ]] || [[ "${ARCH}" == "aarch64" ]]; then
45+
BINARY="${BINARY}-arm64"
3546
fi
3647

3748
# create a temporary directory
@@ -68,4 +79,4 @@ fi
6879

6980
echo "Running buildifier..."
7081
cd "${ROOT_PATH}" || exit
71-
"${BUILDIFIER}" -mode=${MODE} Tiltfile >> "${OUT}" 2>&1
82+
"${BUILDIFIER}" -mode=${MODE} -v Tiltfile >> "${OUT}" 2>&1

‎scripts/aks-as-mgmt.sh

+52-38
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ main() {
9595
fi
9696

9797
create_aks_cluster
98-
set_env_varaibles
98+
set_env_variables
9999
}
100100

101101
create_aks_cluster() {
@@ -197,61 +197,75 @@ create_aks_cluster() {
197197
ASO_CREDENTIAL_SECRET_MODE="podidentity"
198198
}
199199

200-
set_env_varaibles(){
200+
set_env_variables(){
201+
# Ensure temporary files are removed on exit.
202+
trap 'rm -f tilt-settings-temp.yaml tilt-settings-new.yaml combined_contexts.yaml' EXIT
203+
204+
# Create a temporary settings file based on current environment values.
201205
cat <<EOF > tilt-settings-temp.yaml
202-
kustomize_substitutions:
203-
AKS_RESOURCE_GROUP: "${AKS_RESOURCE_GROUP}"
204-
AKS_NODE_RESOURCE_GROUP: "${AKS_NODE_RESOURCE_GROUP}"
206+
aks_as_mgmt_settings:
205207
AKS_MGMT_VNET_NAME: "${AKS_MGMT_VNET_NAME}"
206-
MGMT_CLUSTER_NAME: "${MGMT_CLUSTER_NAME}"
208+
AKS_NODE_RESOURCE_GROUP: "${AKS_NODE_RESOURCE_GROUP}"
209+
AKS_RESOURCE_GROUP: "${AKS_RESOURCE_GROUP}"
210+
APISERVER_LB_DNS_SUFFIX: "${APISERVER_LB_DNS_SUFFIX}"
211+
ASO_CREDENTIAL_SECRET_MODE: "${ASO_CREDENTIAL_SECRET_MODE}"
212+
AZURE_CLIENT_ID: "${AKS_MI_CLIENT_ID}"
207213
AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY: "${AKS_MI_CLIENT_ID}"
214+
AZURE_LOCATION: "${AZURE_LOCATION}"
208215
CI_RG: "${MANAGED_IDENTITY_RG}"
209-
USER_IDENTITY: "${MANAGED_IDENTITY_NAME}"
210216
CLUSTER_IDENTITY_TYPE: "UserAssignedMSI"
211-
ASO_CREDENTIAL_SECRET_MODE: "${ASO_CREDENTIAL_SECRET_MODE}"
217+
CONTROL_PLANE_MACHINE_COUNT: "3"
218+
MGMT_CLUSTER_NAME: "${MGMT_CLUSTER_NAME}"
212219
REGISTRY: "${REGISTRY}"
213-
APISERVER_LB_DNS_SUFFIX: "${APISERVER_LB_DNS_SUFFIX}"
220+
USER_IDENTITY: "${MANAGED_IDENTITY_NAME}"
221+
WORKER_MACHINE_COUNT: "1"
214222
allowed_contexts:
215-
- "$MGMT_CLUSTER_NAME"
223+
- "${MGMT_CLUSTER_NAME}"
216224
- "kind-capz"
217-
azure_location: "${AZURE_LOCATION}"
218225
EOF
219226

220-
# create tilt-settings.yaml if it does not exist
221-
if [ -f tilt-settings.yaml ]; then
222-
echo "tilt-settings.yaml exists"
223-
else
224-
echo "tilt-settings.yaml does not exist, creating one"
225-
touch tilt-settings.yaml
226-
fi
227+
# create tilt-settings.yaml if it does not exist
228+
if [ -f tilt-settings.yaml ]; then
229+
echo "tilt-settings.yaml exists"
230+
else
231+
echo "tilt-settings.yaml does not exist, creating one"
232+
touch tilt-settings.yaml
233+
fi
227234

228-
# copy over the existing allowed_contexts to tilt-settings.yaml if it does not exist
229-
allowed_contexts_exists=$(yq eval '.allowed_contexts' tilt-settings.yaml)
230-
if [ "$allowed_contexts_exists" == "null" ]; then
231-
yq eval '.allowed_contexts = load("tilt-settings-temp.yaml") | .allowed_contexts' tilt-settings-temp.yaml > tilt-settings.yaml
232-
fi
235+
# copy over the existing allowed_contexts to tilt-settings.yaml if it does not exist
236+
allowed_contexts_exists=$(yq eval '.allowed_contexts' tilt-settings.yaml)
237+
if [ "$allowed_contexts_exists" == "null" ]; then
238+
yq eval --inplace '.allowed_contexts = load("tilt-settings-temp.yaml").allowed_contexts' tilt-settings.yaml
239+
fi
240+
241+
# extract allowed_contexts from tilt-settings.yaml
242+
current_contexts=$(yq eval '.allowed_contexts' tilt-settings.yaml | sort -u)
233243

234-
# extract allowed_contexts from tilt-settings.yaml
235-
current_contexts=$(yq eval '.allowed_contexts' tilt-settings.yaml | sort -u)
244+
# extract allowed_contexts from tilt-settings-new.yaml
245+
new_contexts=$(yq eval '.allowed_contexts' tilt-settings-temp.yaml | sort -u)
236246

237-
# extract allowed_contexts from tilt-settings-new.yaml
238-
new_contexts=$(yq eval '.allowed_contexts' tilt-settings-temp.yaml | sort -u)
247+
# combine current and new contexts, keeping the union of both
248+
combined_contexts=$(echo "$current_contexts"$'\n'"$new_contexts" | sort -u)
239249

240-
# combine current and new contexts, keeping the union of both
241-
combined_contexts=$(echo "$current_contexts"$'\n'"$new_contexts" | sort -u)
250+
# create a temporary file since env($combined_contexts) is not supported in yq
251+
echo "$combined_contexts" > combined_contexts.yaml
242252

243-
# create a temporary file since env($combined_contexts) is not supported in yq
244-
echo "$combined_contexts" > combined_contexts.yaml
253+
# update allowed_contexts in tilt-settings.yaml with the combined contexts
254+
yq eval --inplace ".allowed_contexts = load(\"combined_contexts.yaml\")" tilt-settings.yaml
255+
256+
# check if kustomize_substitutions exists, if not add empty one
257+
kustomize_substitutions_exists=$(yq eval '.kustomize_substitutions' tilt-settings.yaml)
258+
if [ "$kustomize_substitutions_exists" == "null" ]; then
259+
yq eval --inplace '.kustomize_substitutions = {}' tilt-settings.yaml
260+
fi
245261

246-
# update allowed_contexts in tilt-settings.yaml with the combined contexts
247-
yq eval --inplace ".allowed_contexts = load(\"combined_contexts.yaml\")" tilt-settings.yaml
262+
# clear out existing aks_as_mgmt_settings before merging new settings
263+
yq eval --inplace 'del(.aks_as_mgmt_settings)' tilt-settings.yaml
248264

249-
# merge the updated kustomize_substitution and azure_location with the existing one in tilt-settings.yaml
250-
yq eval-all 'select(fileIndex == 0) *+ {"kustomize_substitutions": select(fileIndex == 1).kustomize_substitutions, "azure_location": select(fileIndex == 1).azure_location}' tilt-settings.yaml tilt-settings-temp.yaml > tilt-settings-new.yaml
265+
# merge the updated kustomize_substitution and azure_location with the existing one in tilt-settings.yaml
266+
yq eval-all 'select(fileIndex == 0) *+ {"aks_as_mgmt_settings": select(fileIndex == 1).aks_as_mgmt_settings}' tilt-settings.yaml tilt-settings-temp.yaml > tilt-settings-new.yaml
251267

252-
mv tilt-settings-new.yaml tilt-settings.yaml
253-
rm -r combined_contexts.yaml
254-
rm -f tilt-settings-temp.yaml
268+
mv tilt-settings-new.yaml tilt-settings.yaml
255269
}
256270

257271
main

‎scripts/kind-with-registry.sh

+17
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,23 @@ source "${REPO_ROOT}/hack/ensure-tags.sh"
2727
KUBECTL="${REPO_ROOT}/hack/tools/bin/kubectl"
2828
KIND="${REPO_ROOT}/hack/tools/bin/kind"
2929
AZWI="${REPO_ROOT}/hack/tools/bin/azwi"
30+
31+
# Remove aks_as_mgmt_settings from tilt-settings.yaml if it exists
32+
TILT_SETTINGS_FILE="${REPO_ROOT}/tilt-settings.yaml"
33+
if [ -f "$TILT_SETTINGS_FILE" ]; then
34+
# Check if yq is installed
35+
if ! command -v yq &> /dev/null; then
36+
echo "yq is required but not installed. Please install yq first."
37+
exit 1
38+
fi
39+
40+
# Check if aks_as_mgmt_settings exists in the file
41+
if yq e 'has("aks_as_mgmt_settings")' "$TILT_SETTINGS_FILE" | grep -q "true"; then
42+
echo "Removing aks_as_mgmt_settings from tilt-settings.yaml"
43+
yq e 'del(.aks_as_mgmt_settings)' -i "$TILT_SETTINGS_FILE"
44+
fi
45+
fi
46+
3047
AZWI_ENABLED="${AZWI_ENABLED:-true}"
3148
RANDOM_SUFFIX="${RANDOM_SUFFIX:-$(od -An -N4 -tu4 /dev/urandom | tr -d ' ' | head -c 8)}"
3249
export AZWI_STORAGE_ACCOUNT="capzcioidcissuer${RANDOM_SUFFIX}"

0 commit comments

Comments
 (0)