KEP-4330: Change API availability to forward compatibility #5038
Conversation
siyuanfoundation
commented
Jan 14, 2025
•
siyuanfoundation
commented
- One-line PR description: change api availability to forward compatible to make it easier to handle API graduation in compatibility version.
- Issue link: Introduce compatibility version for Kubernetes control-plane upgrades #4330
- Other comments: more discussions can be found in [Compatibility Version] Guidance on API version promotion kubernetes#127791
k8s-ci-robot
added
the
cncf-cla: yes
k8s-ci-robot
added
kind/kep
siyuanfoundation
force-pushed
the
compat-versions-api
branch
2 times, most recently
from
6abb021 to
fbbc5f7
Compare
| Instead, we are proposing to keep forward compatibility of all AIPs in compatibility version mode: | ||
| All APIs group-versions declarations would be modified | ||
| to track which Kubernetes version the API group-versions are removed at. APIs introduced between the emulation version and | ||
| the binary version are still available if the stability level is no less than the APIs introduced before or at the emulation version and enabled at the emulation version. |
There was a problem hiding this comment.
Most of this is referring to API enablement/disablement on a group-version level. However, API lifecycles are generally individual on the group-version-resource level. I think it would make more sense to discuss the individual beta->GA resources rather than the entire group version?
|
|
||
| Alpha APIs may not be enabled in conjunction with emulation version. | ||
|
|
||
| The forward compatibility of APIs should not affect data compatibility because storage version is still controlled by the `MinCompatibilityVersion` regardless of whether the data are created through future versions of the API endpoints. Webhooks should also work fine if the matching policy is `Equivalent`. |
There was a problem hiding this comment.
I think we should add a section on how MinCompatibilityVersion will change storage version since the policy has always been n-1.
There was a problem hiding this comment.
Updated in #5006
| to track which Kubernetes version the API group-versions are introduced (or | ||
| removed) at. | ||
| removed) at. But in practice, that would make code changes of API graduation intractable. |
There was a problem hiding this comment.
intractable only for withholding API introductions. API removals don't need special handling.
There was a problem hiding this comment.
Updated the whole section. Please review again see if it is more clear.
| @@ -610,21 +623,25 @@ The storage version of each group-resource is the newest | |||
|
|
|||
| ### API availability | |||
|
|
|||
| Similar to feature flags, all APIs group-versions declarations will be modified | |||
| Ideally, similar to feature flags, all APIs group-versions declarations should be modified | |||
There was a problem hiding this comment.
Is this referring to the prerelease lifecycle files or a different mechanism? prerelease lifecycle operates on the types instead of the group version.
There was a problem hiding this comment.
it is referring to the the prerelease lifecycle.
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
fbbc5f7 to
ebd2f55
Compare
| removed) at: if an API is introduced after (or removed before) the emulation version, it should not be available at the emulation version. But in practice, that would make code changes of API graduation that wants to use newer APIs intractable. | ||
| For example, if a controller is switching from Beta to GA API at the binary version, a full emulation would mean the whole controller code needs to be |
There was a problem hiding this comment.
My understanding is that we're doing this specifically for controllers. Are there other use cases or should we simplify the wording here and just state up front that we're doing this to support controllers specifically?
| the emulation version is set to. I.e. `--runtime-config=api/<version>` needs | ||
| to be able to turn on APIs exactly like it did for each Kubernetes version that | ||
| emulation version can be set to. | ||
| If an API has GAed and has not been removed at the emulated version, it would be enabled by default at the emulation version. If a newer stable version of the GA API has been introduced between the emulation version and the binary version, the new GA API would also be enabled at the emulation version along with the old GA API. |
There was a problem hiding this comment.
A comparison (maybe in a table), or how emulation works for normal APIs vs how it works for un-emulatable (non-emulatable?) features might make this information easier to digest.
|
|
||
| GA APIs should match the exact set of APIs enabled in GA for the Kubernetes version | ||
| the emulation version is set to. | ||
| Instead, we are proposing to keep forward compatibility of all APIs in compatibility version mode: |
There was a problem hiding this comment.
We're proposing that non-emulatable feature controllers to update to use the newest available API, ignoring emulation API enablement, right? Maybe say this more directly?
There was a problem hiding this comment.
Yes, mainly for controllers for now. May expand to kublet if we want to add compat version in node. Updated the section.
siyuanfoundation
force-pushed
the
compat-versions-api
branch
3 times, most recently
from
f8bce11 to
961b0c9
Compare
siyuanfoundation
changed the title
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
961b0c9 to
9d67760
Compare
|
|
||
| With the `NonEmulatableFeatures` list, the server would fail to start if | ||
| 1. the `EmulationVersion != BinaryVersion` and | ||
| 1. any feature in the `NonEmulatableFeatures` list is disabled by default at the emulation version and explicitly enabled by the `--feature-gates` flag. |
There was a problem hiding this comment.
What alternatives were considered here? It's not clear to me WHY we're deciding to refuse to start apiservers that have a non-emulatable beta on vs. allowing the beta to be turned on but not guaranteeing emulation for the feature.
Having the apiserver refuse to start is a potential footgun for anyone with automation that uses emulation version, and also offers the capability to turn on beta features/APIs.
There was a problem hiding this comment.
Giving more thoughts to it, I think the alternative is the better way to go. Updated the section with pros and cons.
siyuanfoundation
force-pushed
the
compat-versions-api
branch
3 times, most recently
from
69475d9 to
32e11d2
Compare
siyuanfoundation
changed the title
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
32e11d2 to
0cdaece
Compare
siyuanfoundation
force-pushed
the
compat-versions-api
branch
3 times, most recently
from
9e935f8 to
cfafd42
Compare
Signed-off-by: Siyuan Zhang <[email protected]>
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
cfafd42 to
588c1ec
Compare
|
With the addition of /lgtm /assign @jpbetz for approval |
|
@deads2k: GitHub didn't allow me to assign the following users: for, approval. Note that only kubernetes members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
lgtm
|
/approve Thanks all for helping to shape this. |
|
/cc @johnbelamaric @dims for sig-arch approval |
|
@siyuanfoundation: GitHub didn't allow me to request PR reviews from the following users: for, approval. Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
cc @liggitt |
k8s-ci-robot
removed
the
lgtm
Signed-off-by: Siyuan Zhang <[email protected]>
siyuanfoundation
force-pushed
the
compat-versions-api
branch
from
292e647 to
afbb66b
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/approve thanks for the spot check @liggitt |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dims, jpbetz, liggitt, siyuanfoundation The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
