Bug: neonvm-controller reconcile failures are sometimes not retried

[sharnoff]

Environment

Production

Steps to reproduce

Not sure — we typically see this in higher usage regions, although it might just be because that's where the most VMs are running.

Expected result

On failure, VirtualMachine reconcile operations should always be retried with backoff.

Actual result

Sometimes the operation is not retried until the next time the object is modified, which is unbounded.

Other logs, links

• Here's an example where a VM was not re-reconciled for ~4.5h: https://neonprod.grafana.net/goto/HfWG0ApHR?orgId=1
• Also there was prior investigation of the same issue (with different presentation) in https://neondb.slack.com/archives/C08D5352FD0

[stradig]

We hope this might be fixed but Kubernetes upgrade. Waiting for the next Kubernetes upgrade (ref #1264)