KEP-4794: initial proposal for deprecating v1.Endpoints #4975
Conversation
k8s-ci-robot
added
the
sig/network
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: danwinship The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
cncf-cla: yes
danwinship
force-pushed
the
deprecate-endpoints
branch
from
4f695f2 to
85ba04c
Compare
|
I think this makes a lot of sense, we need to find a way to move the ecosystem to EndpointSlices in a way that Endpoints will not even be required ... that I think is a good litmus test. One important thing is that we can not regress on quality, we can not drop just everything that uses Endpoints from e2e or we miss a lot of coverage so SIG Testing should review carefully the test plan cc: @BenTheElder @pohly I also think this is a new challenge in Kubernetes, as v1.Endpoints is a GA API, and some of the proposals like removing them from conformance are tricky or AFAIK we never did it, I also think we have this target of 100% API endpoints coverage that will fail if we remove v1.Endpoints of comformance ... we need SIG Architecture input here @johnbelamaric @dims Independently, definitive a topic we should work toward it, though not one of the highest priority at this point IMHO |
The KEP does not propose to demote the Endpoints API from conformance; it only proposes to demote the Endpoints controller from conformance.
I think you're referring to tests like " |
I'm generally supportive of this direction, but it does raise the question of "what even is an API"? The API would technically still be there, but this change would effectively force any system that had been reading Endpoints managed by the Endpoints controller to transition to the EndpointSlice API which does feel pretty close to a deprecation even if the API does technically still exist. As long as the EndpointSlice Mirroring controller is still running, people will still be able to create Endpoints and have them automatically translated to EndpointSlices, so this is primarily a question of systems that are reading from Endpoints. I do think that we need to have an answer for what we do if/when we migrate functionality from one v1 API to another. In practice, a rather small portion of components are still relying on the Endpoints API, and an even smaller portion are relying on the Endpoints controller. Given that most of our existing e2e test coverage is focused on functionality that is now provided by the EndpointSlice controller, it's possible that a regression in the Endpoints controller could go unnoticed for a very long time. There's also the practical matter that running the Endpoints controller is likely wasting resources on >90% of existing Kubernetes clusters. More clearly communicating that running the Endpoints controller is optional could be a good step here. To refer to the Kubernetes deprecation policy, making the Endpoints controller optional may fall under "Deprecating a feature or behavior", which has the following guidelines:
1 just requires lots of advance notice, and I think it's safe to say that the EndpointSlice controller is more widely tested and likely more reliable than the Endpoints controller at this point. |
Yeah, I think we need to give a lot of heads up, even if we're not using it in SIG-Network owned projects/tools anymore aside from these controller we don't know how users are still relying on the endpoints being populated.
But not actually documented as deprecated, right?
"Clusters with <= 1000 endpoints per service that don't use dual-stack / topology" is probably still a LOT of clusters that may be using it successfully with some other controller / tooling, even if we aren't with kube-proxy? I don't think gateway-api conformance tests are super relevant in this context?
Important distinction that still still seems like a big expectation change, I don't think we have a good precedent for this yet. This seems relevant to @kubernetes/api-approvers in addition to SIG-Arch. |
I had assumed this too, and that's true to the extent that lots of tests depend on the behavior of kube-proxy, and kube-proxy only looks at EndpointSlices. But there are still a surprising number of tests that look at Endpoints explicitly, and even some that look at Endpoints and not EndpointSlices (including, disturbingly, the dual-stack Service tests!) My plan here was to ensure that everything outside of
Right, I should have been clearer in the KEP but I was expecting this might take a while. (Antonio mentioned deprecating the in-tree cloud providers as a comparison point.) On that note, the KEP currently claims that demoting the Endpoints controller tests from conformance is part of "step 1", which I was thinking would sort of constitute the official announcement that we were doing this, but Antonio suggested that maybe it should come later in the process, so providers don't end up disabling the controller while there are still lots of third-party components that aren't ready for it.
Yeah... I had kinda been thinking it was, though it's obvious in hindsight that it's not (I look at those API docs all the time and should have noticed this).
Yes; I was thinking more in terms of "how much third-party software is still using Endpoints rather than EndpointSlices?". Even if most individual users of FooBarProxy™ have small single-stack clusters, the fact that some don't should mean that the authors would have been under pressure to port away from Endpoints for a while now.
I was mentioning it because it means we don't have to worry that there are Gateway implementations that are based on Endpoints rather than EndpointSlice. |
|
Nothing in our stability guarantees stops us from, eg, making and promoting a ValidatingAdmissionPolicy that blocks Endpoints creation. |
+1, I think it should look closer to cloud provider removal:
|
danwinship
force-pushed
the
deprecate-endpoints
branch
from
85ba04c to
e731838
Compare
| ``` | ||
| <<[UNRESOLVED disable-by-default ]>> | ||
|
|
||
| - MAYBE eventually move `endpoints-controller` and |
There was a problem hiding this comment.
Yeah, And even those topics may be seperable. One might still use Endpoints without mirroring?
|
|
||
| ### API Server Warnings | ||
|
|
||
| Whenever any Endpoints object is created or updated (except by the |
There was a problem hiding this comment.
"except..." - do we have enough information to act on the exception? Does it matter?
There was a problem hiding this comment.
We can't recognize it 100% but we can do heuristics. Especially, the endpoints controller sets the endpoints.kubernetes.io/last-change-trigger-time annotation on every Endpoints it creates, and most other Endpoints-creators would not...
|
|
||
| ### Additional Warnings? | ||
|
|
||
| There are other ways we could warn users about Endpoints being deprecated. |
There was a problem hiding this comment.
Can we return warnings on get ?
There was a problem hiding this comment.
No. Strategies can have WarningsOnCreate and WarningsOnUpdate but nothing else
|
|
||
| The obvious problem with this idea is that part of the motivation for | ||
| this is to reduce resource usage, and adding a largish annotation to | ||
| every Endpoints object is taking one step back. (Or does etcd compress |
| every Endpoints object is taking one step back. (Or does etcd compress | ||
| its data in some way that would mitigate this?) | ||
|
|
||
| We could potentially try to have a "virtual" annotation that gets |
There was a problem hiding this comment.
We have some precedent with default-on-read in service and the magic metadata.name label on namespaces.
I'm not super worried about an annotation. Maybe we can make it smaller, but I would not make it a primary decision factor.
e.g. deprecated.k8s.io: The Endpoints API is deprecated, see http://deprecated.k8s.io/endpoints
|
|
||
| We could potentially have `kubectl` print warnings (to stderr) when | ||
| anyone gets or lists endpoints. There doesn't seem to be precedent for | ||
| this, but it seems like it should work. |
There was a problem hiding this comment.
+1 that seems like the most likely place to be seen, IMO
|
|
||
| #### Warnings via logs/metrics/Events | ||
|
|
||
| We could potentially warn within the apiserver about clients Getting, |
There was a problem hiding this comment.
Having a metric seems like a win - counting the number of endpoints API ops or something, but only if the controllers are disabled, otherwise it is noise.
There was a problem hiding this comment.
The metric seems most useful for trying to decide whether you can safely disable the controllers in the first place.
Maybe the metric could be labeled with the service account name of the getter/lister/watcher?
|
|
||
| We have a few options here: | ||
|
|
||
| 1. Do nothing, continue to always generate both an Endpoints and |
There was a problem hiding this comment.
Incremental - I think 1 and 2 are near-term, the rest are deferred decision.
| because a controller _isn't_ enabled seems slightly magical? But | ||
| perhaps there is precedent somewhere else?) | ||
|
|
||
| Another possibility would be to create an `endpoints-cleanup` |
There was a problem hiding this comment.
I prefer this, but it feels sort of permanent
| that were more than 24 hours out of date with respect to their | ||
| EndpointSlices? | ||
|
|
||
| In all cases, we should probably not automatically delete Endpoints |
There was a problem hiding this comment.
Do we track that? I don't think we have an annotation that says so? Maybe we should add that NOW so we can use it later?
There was a problem hiding this comment.
"Endpoints that were not originally created by the Endpoints controller" is not exactly the right concept, I guess.
We should delete all Endpoints that the Endpoints controller itself would have updated/deleted under appropriate circumstances (ie, any Endpoints that corresponds to a Service with a selector). We should not delete Endpoints that the Endpoints controller would have ignored (ie, Endpoints that do not correspond to a Service or that correspond to a selectorless Service).
#4974
/sig network
/cc @aojea @robscott