neonvm-controller: check if deletion timestamp is in the past (#1055)

If we are passed the mark of the deletion timestamp, it means
the deletion is stuck, and we should consider the pod to be failed anyway.

Possible reasons for this are:
1. Node is down.
2. Pod is stuck pulling the image from the container registry.

Signed-off-by: Oleg Vasilev <[email protected]>

Author: Omrigan

pkg/neonvm/controllers/vm_controller.go

@@ -935,6 +935,12 @@ const (
 // This is *similar* to the value of pod.Status.Phase, but we'd like to retain our own abstraction
 // to have more control over the semantics.
 func runnerStatus(pod *corev1.Pod) runnerStatusKind {
+	// Add 5 seconds to account for clock skew and k8s lagging behind.
+	deadline := metav1.NewTime(metav1.Now().Add(-5 * time.Second))
+
+	if pod.DeletionTimestamp != nil && pod.DeletionTimestamp.Before(lo.ToPtr(deadline)) {
+		return runnerFailed
+	}
 	switch pod.Status.Phase {
 	case "", corev1.PodPending:
 		return runnerPending