feat(healthchecks): Adding Agent Health Checks for legacy & hubble control planes

[mereta]

Description

Extending retina port 10093 to serve a /health endpoint.
For subsequent iterations a different checker can be plugged in to the Handler.

The most important changes include modifying the probe paths and parameters, and adding a new health check handler in the server code.

Changes to Kubernetes Deployment Manifests:

• deploy/hubble/manifests/controller/helm/retina/templates/agent/daemonset.yaml: Updated readiness and liveness probe paths from /metrics to /health and added configurable parameters for probe delays, periods, timeouts, failure thresholds, and success thresholds. [1] [2]
• deploy/legacy/manifests/controller/helm/retina/templates/daemonset.yaml: Updated readiness and liveness probe paths from /metrics to /health and added configurable parameters for probe delays, periods, timeouts, failure thresholds, and success thresholds. [1] [2]

Changes to Configuration Values:

• deploy/hubble/manifests/controller/helm/retina/values.yaml: Added configuration parameters for readiness and liveness probes, including initial delay seconds, period seconds, timeout seconds, failure threshold, and success threshold.

Changes to Server Code:

• pkg/server/server.go: Added import for healthz package and implemented serveHealth function to handle /health endpoint using healthz.CheckHandler. [1] [2] [3]

Related Issue

#1047

Checklist

• I have read the contributing documentation.
• I signed and signed-off the commits (git commit -S -s ...). See this documentation on signing commits.
• I have correctly attributed the author(s) of the code.
• I have tested the changes locally.
• I have followed the project's style guidelines.
• I have updated the documentation, if necessary.
• I have added tests, if applicable.

Screenshots (if applicable) or Testing Completed

Testing was completed manually for legacy & hubble control planes.
Windows image tested with legacy retina, hubble retina N/A.

Additional Notes

Add any additional notes or context about the pull request here.

---

Please refer to the CONTRIBUTING.md file for more information on how to contribute to this project.

[vakalapa]

Why create a new port? and not reuse the existing 10093 ? since we are a hostnet: true pod, any port number we use comes with restrictions as it takes away from nodes usable port ranges. In AKS, we will have to pre register the ports we will be using to make sure customers are also aware of the ports they should not use to avoid any conflict.

[mereta]

This seems to be already configured in Legacy & Hubble control planes as default HealthProbeBindAddress.
Its always provided when we set up k8s controller runtime options.
Hubble:

retina/pkg/config/hubble_config_linux.go

Line 42 in 4b12472

HealthProbeBindAddress: ":18001",

Legacy: https://github.com/microsoft/retina/blob/4b12472cc5007e0931238dd5e26f819e6be093b5/cmd/root.go#L42C10-L42C15

I think also @rbtr mentioned having a configurable port.
Do you think we should change to 10093?

[mereta]

Ok I did some testing and got errors when using port 10093.
I don't think we can reuse the Retina port, some findings below:

We start a http api server with that port in controller manager:

retina/pkg/managers/controllermanager/controllermanager.go

Line 56 in b27958f

// create HTTP server for API server

And the trail for setting up health check is:
The HealthProbeBindAddress we pass as part of options in the ctrl manager ends up here where a listener is created.
comment:
This will throw an error if the bind address is invalid or already in use.
(same logic as providing pprof address below)
https://github.com/kubernetes-sigs/controller-runtime/blob/aea2e32a936584b06ae6f7992f856fe7292b0297/pkg/manager/manager.go#L407

If listener exists k8s tries to add HealthProbeServer:
https://github.com/kubernetes-sigs/controller-runtime/blob/aea2e32a936584b06ae6f7992f856fe7292b0297/pkg/manager/internal.go#L400

addHealthProverServer() definition - creates a new server:
https://github.com/kubernetes-sigs/controller-runtime/blob/aea2e32a936584b06ae6f7992f856fe7292b0297/pkg/manager/internal.go#L290

[timraymond]

Yeah, I agree with Vamsi. Controller Manager's implementation isn't fit for purpose here, so we should just use the standard library and write the two HTTP handlers on whatever server is bound to 10093. It will probably provide a better healthcheck overall, since we're testing whether traffic can be served on 10093.

[mereta]

@vakalapa @timraymond Updated the PR.

[github-actions]

This PR will be closed in 7 days due to inactivity.

[ritwikranjan]

Mostly opinions and curious question rather than any explicit suggestion.

[mereta]

@anubhabMajumdar Any input on the values? Do they look ok?