Add a detailed metric for deprovisioning eligible nodes

[hamishforbes]

Tell us about your request

Currently Karpenter exposes karpenter_deprovisioning_eligible_machines as a total count of nodes that are eligible for deprovisioning by deprovisioner type (e.g. consolidation/emptiness)

This is good, I can easily tell that a cluster has nodes that could be deprovisioned.

What's missing is
a) Which nodes?
b) What's blocking deprovisioning (if anything)?

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Ideally I want a dashboard and/or alert that says "Hey there are x nodes in your cluster that could be deprovisioned but aren't because of Y"

I can choose to ignore if that's because they are spots, or because replacing them would take more nodes etc

But I could easily see that if I manually migrate a do-not-evict pod I can free up a node to be consolidated

Are you currently working around this issue?

Manually going through every node (with some educated guessing based on node resource consumption) and running kubectl describe to see why that node is or is not consolidatable

Additional Context

A label on the node would be helpful too / instead, although my preference would be for a metric, if you could at least do kubectl get nodes -l karpenter.sh/deprovisioning-eligible that would be an improvement!

Attachments

No response

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[njtran]

Hey, we're accepting of this if you want to contribute! Here are my thoughts. Feel free to reach out to me on slack if you need.

a) Which nodes?
b) What's blocking deprovisioning (if anything)?

a) If we wanted to add this into the nodes, we would probably surface this through a node-state-metric like the pod-state-metrics.
b) Karpenter surfaces what's blocked as deprovisioning through events. Can you possibly monitor the events in your cluster? Some of the reasons that deprovisioning is blocked is based off the pods on the node. This could come from things like PDBs or do-not-evict pods, where the blocked condition can change at any second if one of the pods/PDBs changes. I worry about the CPU impact on trying to get this right, so we'd need to do some profiling as well to make sure there's no large impact.

Specifically for "Hey there are x nodes in your cluster that could be deprovisioned but aren't because of Y", it's also important to note that it may take a bit to safely deprovision all the nodes that are eligible, which is a tad different from just being blocked as deprovisioning.