Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(healthchecks): Adding Agent Health Checks for legacy & hubble control planes #1092

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

feat(healthchecks): Adding Agent Health Checks for legacy & hubble control planes #1092

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
c7deff8
Adding Example Health Checks
mereta Dec 2, 2024
a1501ce
Basic health checks
mereta Dec 2, 2024
9c6f39b
Basic health checks
mereta Dec 2, 2024
df90b99
Windows images
mereta Dec 2, 2024
a200860
Update cmd/legacy/daemon.go
mereta Dec 4, 2024
e4df974
Applying PR suggestions
mereta Dec 4, 2024
cf30c6c
Windows readiness probe
mereta Dec 4, 2024
755a82b
Merge branch 'main' into healthchecks
mereta Dec 4, 2024
c2dfc17
Merge branch 'main' into healthchecks
mereta Dec 4, 2024
5d2a80e
Merge branch 'main' into healthchecks
mereta Dec 10, 2024
2bf29fa
Merge branch 'main' into healthchecks
mereta Jan 21, 2025
9004b86
Extend retina port to serve /health
mereta Jan 23, 2025
cbcc57d
Undo Verison
mereta Jan 23, 2025
8e06fd1
Extend retina port to serve /health
mereta Jan 23, 2025
201e8c9
Remove defaults
mereta Jan 23, 2025
f897632
Merge branch 'main' into healthchecks
mereta Jan 24, 2025
12dd9c1
Merge branch 'main' into healthchecks
mereta Jan 27, 2025
6abe402
replace port
mereta Jan 29, 2025
01e9fa0
Merge branch 'main' into healthchecks
mereta Jan 29, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 32 additions & 8 deletions deploy/hubble/manifests/controller/helm/retina/templates/agent/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,10 +80,22 @@ spec:
cpu: {{ .Values.resources.limits.cpu | quote }}
readinessProbe:
httpGet:
path: /metrics
port: {{ .Values.agent.container.retina.ports.containerPort }}
initialDelaySeconds: 10
periodSeconds: 30
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
livenessProbe:
httpGet:
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
env:
- name: POD_NAME
valueFrom:
Expand Down Expand Up @@ -203,10 +215,22 @@ spec:
{{- end }}
readinessProbe:
httpGet:
path: /metrics
port: {{ .Values.agent.container.retina.ports.containerPort }}
initialDelaySeconds: 15
periodSeconds: 10
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
livenessProbe:
httpGet:
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
env:
- name: POD_NAME
valueFrom:
Expand Down
26 changes: 26 additions & 0 deletions deploy/hubble/manifests/controller/helm/retina/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -939,3 +939,29 @@ tls:
# -----BEGIN CERTIFICATE-----
# ...
# -----END CERTIFICATE-----
## @param readinessProbe.initialDelaySeconds [array] Initial delay seconds for readinessProbe
## @param readinessProbe.periodSeconds [array] Period seconds for readinessProbe
## @param readinessProbe.timeoutSeconds [array] Timeout seconds for readinessProbe
## @param readinessProbe.failureThreshold [array] Failure threshold for readinessProbe
## @param readinessProbe.successThreshold [array] Success threshold for readinessProbe
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
##
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
successThreshold: 1
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anubhabMajumdar Any input on the values? Do they look ok?

## @param livenessProbe.initialDelaySeconds [array] Initial delay seconds for livenessProbe
## @param livenessProbe.periodSeconds [array] Period seconds for livenessProbe
## @param livenessProbe.timeoutSeconds [array] Timeout seconds for livenessProbe
## @param livenessProbe.failureThreshold [array] Failure threshold for livenessProbe
## @param livenessProbe.successThreshold [array] Success threshold for livenessProbe
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
##
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
successThreshold: 1
45 changes: 27 additions & 18 deletions deploy/standard/manifests/controller/helm/retina/templates/daemonset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,22 +49,22 @@ spec:
- name: {{ include "retina.name" . }}
readinessProbe:
httpGet:
path: /metrics
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds | default "30" }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds | default "30" }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds | default "1" }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold | default "3" }}
successThreshold: {{ .Values.readinessProbe.successThreshold | default "1" }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
livenessProbe:
httpGet:
path: /metrics
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds | default "30" }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds | default "30" }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds | default "1" }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold | default "3" }}
successThreshold: {{ .Values.livenessProbe.successThreshold | default "1" }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
imagePullPolicy: {{ .Values.image.pullPolicy }}
{{- if .Values.daemonset.container.retina.command }}
Expand Down Expand Up @@ -226,15 +226,24 @@ spec:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
readinessProbe:
httpGet:
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.readinessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.readinessProbe.failureThreshold }}
successThreshold: {{ .Values.readinessProbe.successThreshold }}
livenessProbe:
httpGet:
path: /metrics
path: /health
port: {{ .Values.retinaPort }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds | default "30" }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds | default "30" }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds | default "1" }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold | default "3" }}
successThreshold: {{ .Values.livenessProbe.successThreshold | default "1" }}
initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }}
periodSeconds: {{ .Values.livenessProbe.periodSeconds }}
timeoutSeconds: {{ .Values.livenessProbe.timeoutSeconds }}
failureThreshold: {{ .Values.livenessProbe.failureThreshold }}
successThreshold: {{ .Values.livenessProbe.successThreshold }}
securityContext:
capabilities:
add:
Expand Down
24 changes: 12 additions & 12 deletions deploy/standard/manifests/controller/helm/retina/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,25 +173,25 @@ affinity: []
## @param readinessProbe.successThreshold [array] Success threshold for readinessProbe
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
##
readinessProbe: {}
# initialDelaySeconds: 30
# periodSeconds: 10
# timeoutSeconds: 15
# failureThreshold: 5
# successThreshold: 1
readinessProbe:
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
successThreshold: 1
## @param livenessProbe.initialDelaySeconds [array] Initial delay seconds for livenessProbe
## @param livenessProbe.periodSeconds [array] Period seconds for livenessProbe
## @param livenessProbe.timeoutSeconds [array] Timeout seconds for livenessProbe
## @param livenessProbe.failureThreshold [array] Failure threshold for livenessProbe
## @param livenessProbe.successThreshold [array] Success threshold for livenessProbe
## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes
##
livenessProbe: {}
# initialDelaySeconds: 30
# periodSeconds: 10
# timeoutSeconds: 15
# failureThreshold: 5
# successThreshold: 1
livenessProbe:
initialDelaySeconds: 30
periodSeconds: 30
timeoutSeconds: 15
failureThreshold: 3
successThreshold: 1

## @param priorityClassName [string] Indicates the pod's priority
## Ref: https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/
Expand Down
6 changes: 6 additions & 0 deletions pkg/server/server.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import (
"github.com/prometheus/client_golang/prometheus/promhttp"
"go.uber.org/zap"
"golang.org/x/sync/errgroup"
"sigs.k8s.io/controller-runtime/pkg/healthz"
)

type Server struct {
Expand Down Expand Up @@ -44,6 +45,7 @@ func (rt *Server) SetupHandlers() {
exporter.RegisterMetricsServeCallback(func() {
rt.servePrometheusMetrics()
})
rt.serveHealth()
rt.mux.HandleFunc("/debug/pprof/", pprof.Index)
rt.mux.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
rt.mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
Expand All @@ -61,6 +63,10 @@ func (rt *Server) servePrometheusMetrics() {
rt.mux.Get("/metrics", promhttp.HandlerFor(exporter.CombinedGatherer, promhttp.HandlerOpts{}).ServeHTTP)
}

func (rt *Server) serveHealth() {
rt.mux.Get("/health", healthz.CheckHandler{Checker: healthz.Ping}.ServeHTTP)
}

func (rt *Server) Start(ctx context.Context, addr string) error {
srv := &http.Server{Addr: addr, Handler: rt.mux}
g, gctx := errgroup.WithContext(context.Background())
Expand Down
Loading