perf: clear instance type cache after ICE

[jesseanttila-cai]

Fixes #7443

Description

The instanceTypesCache in the InstanceType provider uses a complex cache key that includes the SeqNum of the unavailableOfferings cache. Since all entries of instanceTypesCache become invalid whenever unavailableOfferings is modified, the cache can be flushed in this case to reduce memory usage.

Changes to other parts of the instanceTypesCache key are not considered in this patch. It is likely that a similar issue could be triggered for example by dynamically updating the blockDeviceMappings of a nodeclass based on pod requirements. Since our setup most commonly modifies nodepool/nodeclass configurations in response to ICEs, this patch was sufficient to solve our memory usage issues.

How was this change tested?

A patched version of the Karpenter controller was deployed in a development environment, and memory usage during cluster scale-up has been tracked for a period of two weeks. Peak memory usage appears to be reduced by as much as 80% in situations where several ICEs occur in quick succession, completely eliminating previously seen OOM issues.

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: #
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	aa24e89
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/6759c46af9aec90008ee5316

[github-actions]

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

[coveralls]

Pull Request Test Coverage Report for Build 12281157741

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

• For more information on this, see Tracking coverage changes with pull request builds.
• To avoid this issue with future PRs, see these Recommended CI Configurations.
• For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

• 16 of 16 (100.0%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.04%) to 65.067%

---

Totals
Change from base Build 12265951884:	0.04%
Covered Lines:	5748
Relevant Lines:	8834

---

💛 - Coveralls

[engedaam]

@jesseanttila-cai Can you help me understand why the automatic cache clean-up does not cover the case you are describing? Defined Cache TTL:

• https://github.com/aws/karpenter-provider-aws/blob/main/pkg/operator/operator.go#L172
• https://github.com/aws/karpenter-provider-aws/blob/main/pkg/cache/cache.go
  By default, the instance type provider cache should clean-up the old keys after 6 minutes. Can you share some metrics of the memory difference you saw or a memory profile of the Karpenter controller

[jesseanttila-cai]

@engedaam

By default, the instance type provider cache should clean-up the old keys after 6 minutes

The automatic cache clean-up does indeed limit the effect of the issue, however in practice the number of ICE occurences within the TTL period can be large enough to have a very significant effect on memory usage. The graph shared in #7443 displays this behavior, including the effect of the cache TTL. The heap profile for the flamegraph in the comment is available here.