feat: added NoExecute taint to terminating Nodes

[sadath-12]

Fixes #624 Part 4 (Terminating:NoExecute Taint)

Description

This pr attempts to solve issue of #621 as mentioned #624 . Tainting the nodes as NoExecute before attempting to terminate and also along side tries to make the taints operation code structure a bit more scalable and reusable

How was this change tested?

make presubmit

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[jonathan-innis]

Hey @sadath-12. I'd discuss more on the issue with @njtran about this one. He is driving the design around this issue and is planning to put together an RFC around the implications of this design. This change has a number of implications, once of which is that adding a NoExecute taint necessarily means that pods that don't tolerate the taint will sit on the node up to their tolerationSeconds (default is 5m) up until they are removed. That means that nodes in general will sit on the cluster for 5m longer on average, which is untenable on its own. I think that @njtran has some thoughts around this, but it would be good to discuss this over the issue.

[sadath-12]

Sure,Thanks @jonathan-innis

[njtran]

Added some information on the #629. I think this PR/Design should be explored and iterated on after that is completed, which I believe is the opposite as what you've described.

[sadath-12]

Sure @njtran

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.