Headlamp is not refreshing version 0.280

[sov-msaeed]

when I try to delete resource, it's deleted successfully but headlamp is not refreshing to show the new created pod. the pod is just an example. but it happens with all resources

Environment (please provide info about your environment):

• Installation type: MAC ARM64 (I use macbook M3)
• Headlamp Version: 0.28.0

Are you able to fix this issue?

[Yes (I will propose a PR) / No.]
no

[joaquimrocha]

@skoeva Can you take a look?

[skoeva]

Hi, would you be able to list out the steps to recreate this issue? If there's any specific configuration for your resource(s), any relevant YAML would be useful to see. Any other additional info you think might be helpful would be very much appreciated as well

[illume]

The steps to reproduce are here:

when I try to delete resource, it's deleted successfully but headlamp is not refreshing to show the new created pod. the pod is just an example. but it happens with all resources

[sov-msaeed]

@joaquimrocha is there any update here ? did you try to test it and see if you can reproduce the issue ?

[joaquimrocha]

cc/ @skoeva

[skoeva]

Are you creating a pod with the "Create" button in the sidebar? Would be nice to know how you're creating/deleting resources, that's what I meant before. Sorry if that wasn't clear.

For example, I'm testing on v0.28.0, and using this deployment YAML in the creation dialog from the sidebar:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

I can create and delete it without any issues. I'm seeing similar 502 errors in the console, but those are because I have clusters in Headlamp that aren't actively running. I'm not able to replicate the loading resource errors atm. Do you have the right permissions set up?

[sov-msaeed]

@skoeva the issue is not creating or deleting. if we talk about pod, when i delete it, the console it still shows it's running but it was actually deleted when i check with kubectl or i restart the console. so something is going on with refreshing the status.

[skoeva]

Are you deleting the pod in kubectl or in the UI?

I created and deleted a pod from the UI, and kubectl and the UI both no longer display the deleted pod

[sov-msaeed]

deleting the pod in the ui. in the old version, it was showing already the new pod in the state of starting till it becomes running then the old pod completely deleted without doing anything.
which OS are you using to test ? I am using Mac M3 Probook

[skoeva]

I'm running on WSL but this does not sound like an OS-specific issue. Will see if we can test with an ARM Mac

[giorgiorossi]

I have the same behaviour. Any operation involving a change on the cluster is not updated on the user interface. If you close and restart headlamp application the changes are displayed correctly. Version 0.270 works correctly. For now we are back to version 0.270. S.O. Windows 11.

[joaquimrocha]

@sniok Maybe this is related to the cache from react-query.

[sniok]

could be connected to #2849 and #2796
maybe something with the multiplexer? @knrt10 @illume

[skoeva]

I did suspect it was websocket-related

[sov-msaeed]

@joaquimrocha @skoeva any update here ? did you manage to find the issue ?

[illume]

@sov-msaeed are you able to check the network tab of the web developer console? I’d like to know what response the failed connection gives. For example is if a 500 or 404 and if so what the json it returns (if any).

[illume]

Perhaps it is related to when there is a cluster that is unavailable. I notice in the original screenshot of errors related to clusters that are unavailable.

If you go to the home view in headlamp in other browser tab are there any unavailable clusters? Any that should be available?

I’m wondering if

• the backend is crashing ( headlamp-server process name)
• or if reproducing it relies on having some clusters be unavailable.

Cc @sov-msaeed

From terminal run this to see if the backend is still running:

ps aux | grep headlamp-server

[joaquimrocha]

• the backend is crashing ( headlamp-server process name)

That's a good point. I think we should add a way for the UI to warn the user when something has happened to the backend. So in situations like we can diagnose faster whether this is the issue.

[illume]

@joaquimrocha yes indeed. I created an issue for showing the backend server has crashed in the UI here:

• No user feedback if backend server is not running #2912

(ps. I thought we used to show a banner of some kind? But noticed there was no UI change myself the other day when I accidentally stopped the backend server)

[RobSlgm]

Checked just know with v0.28.1:

• all clusters are available
• deleting a pod doesn't refresh at all (worked with v0.27.x)

but dev tool shows two entries:

Is it somehow related to the version of K8s (i.e. issue #2868)?

[illume]

Ah, that's a good hint. Thanks @RobSlgm I will check that.

[illume]

I tried with minikube start --kubernetes-version=v1.32.0 and couldn't reproduce the issue. Also tried to create a cronjob and see if that caused the problem.

So I don't think the issue is caused by just Kubernetes 1.32.0.

[illume]

@giorgiorossi I wonder if you can help with some information to help us try and reproduce this issue?

• What version of kubernetes?
• Do you have any cronjobs? If so what api version are they made with? (beta or v1?)
• Do you use an ingress? If so which?

[illume]

For anyone having the issue, can you please check the developer console for this?

websocket still open?

The websocket should have a message like this showing it's open.

messages arriving?

There should be messages coming through in the messages tab.
After deleting a pod there should be a new message arriving.

[RobSlgm]

Hi @illume

While opening Workloads > Pods these messages arrive

{"clusterId":"testlab","path":"/api/v1/namespaces","query":"","userId":"","data":"{\"state\":\"error\",\"error\":\"dialing WebSocket: websocket: bad handshake\"}","type":"STATUS"}

{"clusterId":"testlab","error":"dialing WebSocket: websocket: bad handshake"}

{"clusterId":"testlab","path":"/api/v1/pods","query":"","userId":"","data":"{\"state\":\"error\",\"error\":\"dialing WebSocket: websocket: bad handshake\"}","type":"STATUS"}

While deleting a pod -> nothing happens.

[illume]

btw, these are the avenues I'm looking to reproduce with tomorrow:

• https and ws vs wss when using https via an ingress or port forward
• firefox and above

[giorgiorossi]

Hi, Sorry for the delayed reply.
With headlamp v 0.28 and 0.29.1 I have the same behaviour, no refresh.

Kubernetes v 1.27.16 and 1.31.1
No cronjobs
We use a kubernetes installation provided by our cloud provider that exposes the cluster remote access api

I made a another test. After installing 0.28.1, I installed 0.27, without first uninstalling 0.28.1. In this case, 0.27 also does not refresh.
The same test with 0.28 and 0.27 instead allowed 0.27 to work correctly.
By uninstalling the app from Windows and reinstalling 0.27 everything works correctly.