Skip to content

Commit

Permalink
feat: CSE-based bootstrapping with bootstrapping client mode (#527)
Browse files Browse the repository at this point in the history
* chore: update devcontainer go version

* chore: refresh toolcain

* chore: additional processing on verify
(and migration to kube-system)

* chore: bump dependencies

* chore: refresh Helm charts

* chore: update golangci config

* chore: remove feature gate for drift

* chore: update pre-commit tooling

* chore: update the shape of main

* chore: update the alt operator

* chore: update the API (move kubelet config to AKSNodeClass)

* chore: migrate cloud provider to v1 API

* chore: migrate operator to v1 API

* chore: migrate controllers to v1 API

* chore: add nodeclass status controller

* chore: migrate providers to v1 API

* chore: migrate test pkg to v1 API

* chore: update utils

* chore: update and migrate E2E tests to v1 API

* feat: refresh and relink CRDs

* fix: move code generation into subfolders to fix golangci-lint

(typecheck detecting multiple main.go)

* fix: enable most of govet in golangci

* fix(linting): exclude alt operator logger

* fix: add nodeclass termination controller

* fix(lint): restore linting on verify

* feat: add nodeclass hash controller

* fix: register additional nodeclass and status controllers

* fix(e2e): better selection of karpenter pod for logs

* fix(e2e): fix utilization suite

* chore(e2e): add events to dump-logs (and simplify)

* chore: rename v1 to corev1

* fix: remove extra $

* fix(e2e): add cilium label and taint

* fix(e2e): fix labels and disruption for deamonset test

* feat: update kubelet configuration

* fix: conflicting nodeclaim.garbagecollcation controller name

* chore: restore webhooks in alt operator

* Clean up commented out webhook code

* chore: swagger spec for nodebootstrapping GET and autogenerated client

* feat: bootstrapping client provision mode

* chore: general code improvements

* fix: required VnetCidrs wasn't populated

* chore: read error response as text

* fix: handle the issue of mismatched error format in response body/header

* chore: more accurate overhead subtraction for instancetype

* fix: correct VM memory passing

* test: some unit tests and improvements

* chore: linter changes and small improvements

* feat: makefile command for swagger client generation

* chore: remove unused parameters in readResponse()

* fix: swagger binary not found

* chore: small improvements

* chore: added missing fullstops in other env descriptions

* chore: rename bootstrapping types to scriptless and customscripts to align with AgentBaker

* fix(test): fix test for credential provider URL in custom data

* Make webhooks work in AKS CCP context (#537)

This requires quite a bit of hacking, mostly overriding certain things
in the ctx. The major items are:

 * Copy and modify knative/pkg/webhook/resourcesemantics/conversion to
   support CRD clientConfig.url in addition to clientConfig.service.
 * Copy and modify karpenter/pkg/webhooks/webhooks.go to support
   overriding the informer factory, so that we can point it at the
   CCP APIServer rather than overlay.
 * Override Start and supporting methods on the provider specific
   operator in pkg/operator/operator.go to allow invoking our modified
   version of karpenter/pkg/webhooks/webhooks.go.

* chore: MaxPods is already defaulted

* chore: remove failSwapOn from kubelet settings in AKSNodeClass

* fix: populate nodeClaim.Status.ImageID

* fix: record NodeClass hash and add drift on static fields

* chore: rename variabled

* fix: remove outdated comment

* fix: typo

* chore: update CRDs

* feat: support custom kubeletconfig per v1

* chore: ignore false positive golint overflow check

* test: fix option for provision mode in testing

---------

Co-authored-by: tallaxes <[email protected]>
Co-authored-by: Matthew Christopher <[email protected]>
  • Loading branch information
3 people authored Oct 25, 2024
1 parent 46fc0d3 commit 4f922e8
Show file tree
Hide file tree
Showing 49 changed files with 4,380 additions and 68 deletions.
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ coverage:
go tool cover -html coverage.out -o coverage.html

verify: toolchain tidy download ## Verify code. Includes dependencies, linting, formatting, etc
make az-swagger-generate-clients-raw
go generate ./...
hack/boilerplate.sh
cp $(KARPENTER_CORE_DIR)/pkg/apis/crds/* pkg/apis/crds
Expand Down
8 changes: 8 additions & 0 deletions Makefile-az.mk
Original file line number Diff line number Diff line change
Expand Up @@ -358,3 +358,11 @@ az-helm-install-snapshot: az-configure-values ## Install Karpenter snapshot rele

az-rmcrds: ## Delete Karpenter CRDs
kubectl delete crd nodepools.karpenter.sh nodeclaims.karpenter.sh aksnodeclasses.karpenter.azure.com

az-swagger-generate-clients-raw:
cd pkg/provisionclients && swagger generate client -f swagger/*.json
hack/azure/temp_fix_get_bootstrapping_resp_error.sh

az-swagger-generate-clients: az-swagger-generate-clients-raw
hack/boilerplate.sh
make tidy
14 changes: 13 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@ require (
github.com/blang/semver/v4 v4.0.0
github.com/go-logr/logr v1.4.2
github.com/go-logr/zapr v1.3.0
github.com/go-openapi/errors v0.22.0
github.com/go-openapi/runtime v0.28.0
github.com/go-openapi/strfmt v0.23.0
github.com/go-openapi/swag v0.23.0
github.com/go-openapi/validate v0.24.0
github.com/go-playground/validator/v10 v10.22.1
github.com/google/go-cmp v0.6.0
github.com/imdario/mergo v0.3.16
Expand Down Expand Up @@ -68,6 +73,7 @@ require (
github.com/Azure/go-autorest/logger v0.2.1 // indirect
github.com/Azure/go-autorest/tracing v0.6.0 // indirect
github.com/AzureAD/microsoft-authentication-library-for-go v1.2.2 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/avast/retry-go v3.0.0+incompatible // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blendle/zapdriver v1.3.1 // indirect
Expand All @@ -83,9 +89,11 @@ require (
github.com/go-kit/log v0.2.1 // indirect
github.com/go-logfmt/logfmt v0.6.0 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-openapi/analysis v0.23.0 // indirect
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/go-openapi/loads v0.22.0 // indirect
github.com/go-openapi/spec v0.21.0 // indirect
github.com/go-playground/locales v0.14.1 // indirect
github.com/go-playground/universal-translator v0.18.1 // indirect
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
Expand All @@ -108,9 +116,12 @@ require (
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/leodido/go-urn v1.4.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mitchellh/mapstructure v1.5.0 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/oklog/ulid v1.3.1 // indirect
github.com/opentracing/opentracing-go v1.2.0 // indirect
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.0 // indirect
Expand All @@ -122,6 +133,7 @@ require (
github.com/shopspring/decimal v1.3.1 // indirect
github.com/spf13/cobra v1.8.0 // indirect
github.com/spf13/pflag v1.0.5 // indirect
go.mongodb.org/mongo-driver v1.14.0 // indirect
go.opencensus.io v0.24.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
Expand Down
26 changes: 26 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,8 @@ github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d/go.mod h1:rBZYJk5
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137 h1:s6gZFSlWYmbqAuRjVTiNNhvNRfY2Wxp9nhfyel4rklc=
github.com/alecthomas/units v0.0.0-20211218093645-b94a6e3cc137/go.mod h1:OMCwj8VM1Kc9e19TLln2VL61YJF0x1XFtfdL4JdbSyE=
github.com/antihax/optional v1.0.0/go.mod h1:uupD/76wgC+ih3iEmQUL+0Ugr19nfwCT1kdvxnR2qWY=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2/go.mod h1:WaHUgvxTVq04UNunO+XhnAqY/wQc+bxr74GqbsZ/Jqw=
github.com/avast/retry-go v3.0.0+incompatible h1:4SOWQ7Qs+oroOTQOYnAHqelpCO0biHSxpiH9JdtuBj0=
github.com/avast/retry-go v3.0.0+incompatible/go.mod h1:XtSnn+n/sHqQIpZ10K1qAevBhOOCWBLXXy3hyiqqBrY=
github.com/awslabs/operatorpkg v0.0.0-20240805231134-67d0acfb6306 h1:0dzaVod1XLEc38H4IB+KOgStoCt8RkCVI4t+XsSPrWE=
Expand Down Expand Up @@ -190,12 +192,26 @@ github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/go-logr/zapr v1.3.0 h1:XGdV8XW8zdwFiwOA2Dryh1gj2KRQyOOoNmBy4EplIcQ=
github.com/go-logr/zapr v1.3.0/go.mod h1:YKepepNBd1u/oyhd/yQmtjVXmm9uML4IXUgMOwR8/Gg=
github.com/go-openapi/analysis v0.23.0 h1:aGday7OWupfMs+LbmLZG4k0MYXIANxcuBTYUC03zFCU=
github.com/go-openapi/analysis v0.23.0/go.mod h1:9mz9ZWaSlV8TvjQHLl2mUW2PbZtemkE8yA5v22ohupo=
github.com/go-openapi/errors v0.22.0 h1:c4xY/OLxUBSTiepAg3j/MHuAv5mJhnf53LLMWFB+u/w=
github.com/go-openapi/errors v0.22.0/go.mod h1:J3DmZScxCDufmIMsdOuDHxJbdOGC0xtUynjIx092vXE=
github.com/go-openapi/jsonpointer v0.21.0 h1:YgdVicSA9vH5RiHs9TZW5oyafXZFc6+2Vc1rr/O9oNQ=
github.com/go-openapi/jsonpointer v0.21.0/go.mod h1:IUyH9l/+uyhIYQ/PXVA41Rexl+kOkAPDdXEYns6fzUY=
github.com/go-openapi/jsonreference v0.21.0 h1:Rs+Y7hSXT83Jacb7kFyjn4ijOuVGSvOdF2+tg1TRrwQ=
github.com/go-openapi/jsonreference v0.21.0/go.mod h1:LmZmgsrTkVg9LG4EaHeY8cBDslNPMo06cago5JNLkm4=
github.com/go-openapi/loads v0.22.0 h1:ECPGd4jX1U6NApCGG1We+uEozOAvXvJSF4nnwHZ8Aco=
github.com/go-openapi/loads v0.22.0/go.mod h1:yLsaTCS92mnSAZX5WWoxszLj0u+Ojl+Zs5Stn1oF+rs=
github.com/go-openapi/runtime v0.28.0 h1:gpPPmWSNGo214l6n8hzdXYhPuJcGtziTOgUpvsFWGIQ=
github.com/go-openapi/runtime v0.28.0/go.mod h1:QN7OzcS+XuYmkQLw05akXk0jRH/eZ3kb18+1KwW9gyc=
github.com/go-openapi/spec v0.21.0 h1:LTVzPc3p/RzRnkQqLRndbAzjY0d0BCL72A6j3CdL9ZY=
github.com/go-openapi/spec v0.21.0/go.mod h1:78u6VdPw81XU44qEWGhtr982gJ5BWg2c0I5XwVMotYk=
github.com/go-openapi/strfmt v0.23.0 h1:nlUS6BCqcnAk0pyhi9Y+kdDVZdZMHfEKQiS4HaMgO/c=
github.com/go-openapi/strfmt v0.23.0/go.mod h1:NrtIpfKtWIygRkKVsxh7XQMDQW5HKQl6S5ik2elW+K4=
github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+GrE=
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-openapi/validate v0.24.0 h1:LdfDKwNbpB6Vn40xhTdNZAnfLECL81w+VX3BumrGD58=
github.com/go-openapi/validate v0.24.0/go.mod h1:iyeX1sEufmv3nPbBdX3ieNviWnOZaJ1+zquzJEf2BAQ=
github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s=
github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4=
github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA=
Expand Down Expand Up @@ -339,6 +355,8 @@ github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJ
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4=
github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE=
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
Expand All @@ -350,10 +368,14 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/oklog/ulid v1.3.1 h1:EGfNDEx6MqHz8B3uNV6QAib1UR2Lm97sHi3ocA6ESJ4=
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
github.com/onsi/ginkgo/v2 v2.20.2 h1:7NVCeyIWROIAheY21RLS+3j2bb52W0W82tkberYytp4=
github.com/onsi/ginkgo/v2 v2.20.2/go.mod h1:K9gyxPIlb+aIvnZ8bd9Ak+YP18w3APlR+5coaZoE2ag=
github.com/onsi/gomega v1.34.2 h1:pNCwDkzrsv7MS9kpaQvVb1aVLahQXyJ/Tv5oAZMI3i8=
github.com/onsi/gomega v1.34.2/go.mod h1:v1xfxRgk0KIsG+QOdm7p8UosrOzPYRo60fd3B/1Dukc=
github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+1B0VhjKrZUs=
github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
github.com/patrickmn/go-cache v2.1.0+incompatible h1:HRMgzkcYKYpi3C8ajMPV8OFXaaRUnok+kx1WdO15EQc=
github.com/patrickmn/go-cache v2.1.0+incompatible/go.mod h1:3Qf8kWWT7OJRJbdiICTKqZju1ZixQ/KpMGzzAfe6+WQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
Expand Down Expand Up @@ -440,6 +462,8 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.mongodb.org/mongo-driver v1.14.0 h1:P98w8egYRjYe3XDjxhYJagTokP/H6HzlsnojRgZRd80=
go.mongodb.org/mongo-driver v1.14.0/go.mod h1:Vzb0Mk/pa7e6cWw85R4F/endUC3u0U9jGcNU603k65c=
go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU=
go.opencensus.io v0.22.0/go.mod h1:+kGneAE2xo2IficOXnaByMWTGM9T73dGwxeWcUqIpI8=
go.opencensus.io v0.22.2/go.mod h1:yxeiOL68Rb0Xd1ddK5vPZ/oVn4vY4Ynel7k9FzqtOIw=
Expand All @@ -454,6 +478,8 @@ go.opentelemetry.io/otel v1.24.0 h1:0LAOdjNmQeSTzGBzduGe/rU4tZhMwL5rWgtp9Ku5Jfo=
go.opentelemetry.io/otel v1.24.0/go.mod h1:W7b9Ozg4nkF5tWI5zsXkaKKDjdVjpD4oAt9Qi/MArHo=
go.opentelemetry.io/otel/metric v1.24.0 h1:6EhoGWWK28x1fbpA4tYTOWBkPefTDQnb8WSGXlc88kI=
go.opentelemetry.io/otel/metric v1.24.0/go.mod h1:VYhLe1rFfxuTXLgj4CBiyz+9WYBA8pNGJgDcSFRKBco=
go.opentelemetry.io/otel/sdk v1.24.0 h1:YMPPDNymmQN3ZgczicBY3B6sf9n62Dlj9pWD3ucgoDw=
go.opentelemetry.io/otel/sdk v1.24.0/go.mod h1:KVrIYw6tEubO9E96HQpcmpTKDVn9gdv35HoYiQWGDFg=
go.opentelemetry.io/otel/trace v1.24.0 h1:CsKnnL4dUAr/0llH9FKuc698G04IrpWV0MQA/Y1YELI=
go.opentelemetry.io/otel/trace v1.24.0/go.mod h1:HPc3Xr/cOApsBI154IU0OI0HJexz+aw5uPdbs3UCjNU=
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
Expand Down
15 changes: 15 additions & 0 deletions hack/azure/temp_fix_get_bootstrapping_resp_error.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/bin/bash

# Define the file path
FILE="pkg/provisionclients/client/operations/node_bootstrapping_get_responses.go"

# Check if the file exists
if [ ! -f "$FILE" ]; then
echo "File $FILE does not exist."
exit 1
fi

# Use sed to delete the readResponse() method if it exists
sed -i '/func (o \*NodeBootstrappingGetDefault) readResponse/,/^}/d' "$FILE"

echo "readResponse() method deleted from $FILE if it existed. This is for a temporary fix that is in node_bootstrapping_get_responses_override.go."
1 change: 1 addition & 0 deletions hack/toolchain.sh
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ tools() {
go install github.com/rhysd/actionlint/cmd/[email protected]
go install github.com/mattn/[email protected]
go install github.com/google/go-containerregistry/cmd/[email protected]
go install github.com/go-swagger/go-swagger/cmd/[email protected]

if ! echo "$PATH" | grep -q "${GOPATH:-undefined}/bin\|$HOME/go/bin"; then
echo "Go workspace's \"bin\" directory is not in PATH. Run 'export PATH=\"\$PATH:\${GOPATH:-\$HOME/go}/bin\"'."
Expand Down
3 changes: 3 additions & 0 deletions pkg/consts/consts.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,4 +28,7 @@ const (
NetworkDataplaneAzure = "azure"

DefaultKubernetesMaxPods = 250

ProvisionModeAKSScriptless = "aksscriptless"
ProvisionModeBootstrappingClient = "bootstrappingclient"
)
3 changes: 3 additions & 0 deletions pkg/operator/operator.go
Original file line number Diff line number Diff line change
Expand Up @@ -117,10 +117,12 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
options.FromContext(ctx).ClusterEndpoint,
azConfig.TenantID,
azConfig.SubscriptionID,
azConfig.ResourceGroup,
azConfig.KubeletIdentityClientID,
azConfig.NodeResourceGroup,
azConfig.Location,
vnetGUID,
options.FromContext(ctx).ProvisionMode,
)
instanceTypeProvider := instancetype.NewDefaultProvider(
azConfig.Location,
Expand All @@ -143,6 +145,7 @@ func NewOperator(ctx context.Context, operator *operator.Operator) (context.Cont
azConfig.Location,
azConfig.NodeResourceGroup,
azConfig.SubscriptionID,
options.FromContext(ctx).ProvisionMode,
)

return ctx, &Operator{
Expand Down
12 changes: 8 additions & 4 deletions pkg/operator/options/options.go
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,9 @@ type Options struct {
SubnetID string // => VnetSubnetID to use (for nodes in Azure CNI Overlay and Azure CNI + pod subnet; for for nodes and pods in Azure CNI), unless overridden via AKSNodeClass
setFlags map[string]bool

NodeResourceGroup string
NodeResourceGroup string
ProvisionMode string
NodeBootstrappingServerURL string
}

func (o *Options) AddFlags(fs *coreoptions.FlagSet) {
Expand All @@ -85,12 +87,14 @@ func (o *Options) AddFlags(fs *coreoptions.FlagSet) {
fs.StringVar(&o.KubeletClientTLSBootstrapToken, "kubelet-bootstrap-token", env.WithDefaultString("KUBELET_BOOTSTRAP_TOKEN", ""), "[REQUIRED] The bootstrap token for new nodes to join the cluster.")
fs.StringVar(&o.SSHPublicKey, "ssh-public-key", env.WithDefaultString("SSH_PUBLIC_KEY", ""), "[REQUIRED] VM SSH public key.")
fs.StringVar(&o.NetworkPlugin, "network-plugin", env.WithDefaultString("NETWORK_PLUGIN", consts.NetworkPluginAzure), "The network plugin used by the cluster.")
fs.StringVar(&o.NetworkPluginMode, "network-plugin-mode", env.WithDefaultString("NETWORK_PLUGIN_MODE", consts.NetworkPluginModeOverlay), "network plugin mode of the cluster")
fs.StringVar(&o.NetworkPluginMode, "network-plugin-mode", env.WithDefaultString("NETWORK_PLUGIN_MODE", consts.NetworkPluginModeOverlay), "network plugin mode of the cluster.")
fs.StringVar(&o.NetworkPolicy, "network-policy", env.WithDefaultString("NETWORK_POLICY", ""), "The network policy used by the cluster.")
fs.StringVar(&o.NetworkDataplane, "network-dataplane", env.WithDefaultString("NETWORK_DATAPLANE", "cilium"), "The network dataplane used by the cluster.")
fs.StringVar(&o.SubnetID, "vnet-subnet-id", env.WithDefaultString("VNET_SUBNET_ID", ""), "The default subnet ID to use for new nodes. This must be a valid ARM resource ID for subnet that does not overlap with the service CIDR or the pod CIDR")
fs.StringVar(&o.SubnetID, "vnet-subnet-id", env.WithDefaultString("VNET_SUBNET_ID", ""), "The default subnet ID to use for new nodes. This must be a valid ARM resource ID for subnet that does not overlap with the service CIDR or the pod CIDR.")
fs.Var(newNodeIdentitiesValue(env.WithDefaultString("NODE_IDENTITIES", ""), &o.NodeIdentities), "node-identities", "User assigned identities for nodes.")
fs.StringVar(&o.NodeResourceGroup, "node-resource-group", env.WithDefaultString("AZURE_NODE_RESOURCE_GROUP", ""), "[REQUIRED] the resource group created and managed by AKS where the nodes live")
fs.StringVar(&o.NodeResourceGroup, "node-resource-group", env.WithDefaultString("AZURE_NODE_RESOURCE_GROUP", ""), "[REQUIRED] the resource group created and managed by AKS where the nodes live.")
fs.StringVar(&o.ProvisionMode, "provision-mode", env.WithDefaultString("PROVISION_MODE", consts.ProvisionModeAKSScriptless), "[UNSUPPORTED] The provision mode for the cluster.")
fs.StringVar(&o.NodeBootstrappingServerURL, "nodebootstrapping-server-url", env.WithDefaultString("NODEBOOTSTRAPPING_SERVER_URL", ""), "[UNSUPPORTED] The url for the node bootstrapping provider server.")
}

func (o Options) GetAPIServerName() string {
Expand Down
13 changes: 13 additions & 0 deletions pkg/operator/options/options_validation.go
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ func (o Options) Validate() error {
o.validateNetworkingOptions(),
o.validateVMMemoryOverheadPercent(),
o.validateVnetSubnetID(),
o.validateProvisionMode(),
validate.Struct(o),
)
}
Expand Down Expand Up @@ -83,6 +84,18 @@ func (o Options) validateVMMemoryOverheadPercent() error {
return nil
}

func (o Options) validateProvisionMode() error {
if o.ProvisionMode != consts.ProvisionModeAKSScriptless && o.ProvisionMode != consts.ProvisionModeBootstrappingClient {
return fmt.Errorf("provision-mode is invalid: %s", o.ProvisionMode)
}
if o.ProvisionMode == consts.ProvisionModeBootstrappingClient {
if o.NodeBootstrappingServerURL == "" {
return fmt.Errorf("nodebootstrapping-server-url is required when provision-mode is bootstrappingclient")
}
}
return nil
}

func (o Options) validateRequiredFields() error {
if o.ClusterEndpoint == "" {
return fmt.Errorf("missing field, cluster-endpoint")
Expand Down
28 changes: 28 additions & 0 deletions pkg/operator/options/suite_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ var _ = Describe("Options", func() {
"NETWORK_PLUGIN",
"NETWORK_POLICY",
"NODE_IDENTITIES",
"PROVISION_MODE",
"NODEBOOTSTRAPPING_SERVER_URL",
}

var fs *coreoptions.FlagSet
Expand Down Expand Up @@ -95,6 +97,8 @@ var _ = Describe("Options", func() {
os.Setenv("NETWORK_POLICY", "env-network-policy")
os.Setenv("NODE_IDENTITIES", "/subscriptions/1234/resourceGroups/mcrg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/envid1,/subscriptions/1234/resourceGroups/mcrg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/envid2")
os.Setenv("VNET_SUBNET_ID", "/subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/sillygeese/providers/Microsoft.Network/virtualNetworks/karpentervnet/subnets/karpentersub")
os.Setenv("PROVISION_MODE", "bootstrappingclient")
os.Setenv("NODEBOOTSTRAPPING_SERVER_URL", "https://nodebootstrapping-server-url")
fs = &coreoptions.FlagSet{
FlagSet: flag.NewFlagSet("karpenter", flag.ContinueOnError),
}
Expand All @@ -112,6 +116,8 @@ var _ = Describe("Options", func() {
NetworkPolicy: lo.ToPtr("env-network-policy"),
SubnetID: lo.ToPtr("/subscriptions/12345678-1234-1234-1234-123456789012/resourceGroups/sillygeese/providers/Microsoft.Network/virtualNetworks/karpentervnet/subnets/karpentersub"),
NodeIdentities: []string{"/subscriptions/1234/resourceGroups/mcrg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/envid1", "/subscriptions/1234/resourceGroups/mcrg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/envid2"},
ProvisionMode: lo.ToPtr("bootstrappingclient"),
NodeBootstrappingServerURL: lo.ToPtr("https://nodebootstrapping-server-url"),
}))
})
})
Expand Down Expand Up @@ -273,6 +279,28 @@ var _ = Describe("Options", func() {
)
Expect(err).ToNot(HaveOccurred())
})
It("should fail validation when ProvisionMode is not valid", func() {
err := opts.Parse(
fs,
"--cluster-name", "my-name",
"--cluster-endpoint", "https://karpenter-000000000000.hcp.westus2.staging.azmk8s.io",
"--kubelet-bootstrap-token", "flag-bootstrap-token",
"--ssh-public-key", "flag-ssh-public-key",
"--provision-mode", "ekeselfexposed",
)
Expect(err).To(MatchError(ContainSubstring("provision-mode")))
})
It("should fail validation when ProvisionMode is bootstrappingclient but NodebootstrappingServerURL is not provided", func() {
err := opts.Parse(
fs,
"--cluster-name", "my-name",
"--cluster-endpoint", "https://karpenter-000000000000.hcp.westus2.staging.azmk8s.io",
"--kubelet-bootstrap-token", "flag-bootstrap-token",
"--ssh-public-key", "flag-ssh-public-key",
"--provision-mode", "bootstrappingclient",
)
Expect(err).To(MatchError(ContainSubstring("nodebootstrapping-server-url")))
})
})
})

Expand Down
Loading

0 comments on commit 4f922e8

Please sign in to comment.