GEP-1713: ListenerSets - Standard Mechanism to Merge Gateway Listeners (rev 2)

[dprotaso]

This replaces #1863
What type of PR is this?

/kind gep

What this PR does / why we need it:

Outlines a mechanism to merge Gateway Listeners

Which issue(s) this PR fixes:

Fixes #1713

Does this PR introduce a user-facing change?:

NONE

[youngnick]

Quite a few comments here, thanks for keeping working on pushing this forward.

[robscott]

Thanks for all the work on this @dprotaso!

[youngnick]

This LGTM for Provisional (or maybe even Implementable, since we'll need to implement it and try it out) now.

[robscott]

Status update: We're at the end of the 2-day extension for this GEP to make it into this release, so unfortunately I think this is going to have to switch to targeting v1.3. Fortunately, this new shorter release cycle means that's actually not that far away.

Ultimately, I think you're right. For this to make sense, we'd need some structural changes to Gateway, particularly making Listeners optional. That's a huge change to a GA API (even if it would initially be limited to experimental channel, breaking changes to something labeled as "v1"can be confusing). Combined with the huge scope of the PR, and most of the feedback just coming from Nick and I, I think this particular GEP would benefit from some more soak time for additional reviewers + to smooth out any areas we're uncertain. Unfortunately, I think it's just too big to try to squeeze in today.

With that said, I would like to see this GEP make it in early in the next release cycle (similar to how named Route rules just barely missed the previous cycle but was merged at the beginning of this one).

I think it would be useful to do the following between now and then:

1. Continue to iterate on this GEP, discussing in meetings as necessary (this can include merging this as provisional with some unresolved sections and iterating in smaller PRs)
2. Build out a clearer vision for how this can be expanded to cover more use cases in the future (particularly the ones we discussed in the hierarchy doc) - that will help make a stronger case for why this is important
3. Present these ideas at the SIG-Network review session for v1.2 so we can get early feedback on this overall idea and adjust the proposal accordingly
4. Warn all controller authors that it's possible that Listeners may become optional in a future release of Gateway API and they should make sure they can handle that case

Thanks again for all the work you've been putting in to push this forward @dprotaso! I really do appreciate it.

[dprotaso]

/retest

[robscott]

Thanks for all the work on this @dprotaso! Took a fresh pass and I think this largely makes sense.

[robscott]

Thanks for capturing this!

+1 to past me on all points 🙃. I'd argue that we should probably make attempts at resolving both 1 and 2 here.

[youngnick]

For multiple hostnames, I can see the utility, but it risks making the rules about Listener -> Route hostname matching even more complicated.

[dprotaso]

Optional port would be nice - then let the implementation choose it. Though it's a rare use case for me.

[robscott]

My votes on the above:

1. Yes
2. No
3. Not that I can think of

[robscott]

Would like to get some resolution on #3497 before we release this API, as I'd prefer to switch this group to gateway.networking.x-k8s.io

[robscott]

Although we can certainly revisit this in the API phase, I think I can safely say that a majority of maintainers are in favor of using the new experimental API group for this.

[dprotaso]

I think I can safely say that a majority of maintainers are in favor of using the new experimental API group for this.

The discussion #3497 seems to be leaning the other way - was there a meeting/vote somewhere?

[robscott]

There's been some renewed interest in pushing L4 + ClusterIP Gateways forward, and one thing that could help with that is if Listeners could point directly to backends. I'd personally like to explore that in this GEP as a follow up to the similar idea in the doc. (No need to take action here until we get some more comments/feedback on this idea).

[mlavacca]

I think that including L4 considerations at the very first stage of the design, before having the actual API in place is too much of a stretch. Getting rid of L4 routes in favor of including L4 routing information (i.e., backends) is a controversial one (for which I am on the fence honestly) and I don't think that should block this GEP which has been in the work for a long time and needs to be moved forward.

Putting here L4 will likely require a lot of discussion, partially off-topic with the broader goal of this GEP.

[youngnick]

Reading back over this, I think the main things outstanding are:

• Will we start this off with the separate experimental API group (as per New Experimental APIs should start in x-k8s.io API Group #3497)?
• Will we make port optional in the ListenerSet version of Listener?
• Will we keep the parentRef as a list or move it back to a singular one?

[robscott]

• Will we start this off with the separate experimental API group (as per New Experimental APIs should start in x-k8s.io API Group #3497)?

I'm strongly in favor of this, and the idea seems to be gaining some traction on the corresponding issue.

• Will we make port optional in the ListenerSet version of Listener?

I would like to ensure that this has a well documented path forward to supporting all ports on a single Listener. That could either be a port range or an empty port. We don't have to start with either of those, I'd just like to have a documented plan for how we could get there.

• Will we keep the parentRef as a list or move it back to a singular one?

I'm on the fence on this one, left a comment above.

[mikemorris]

I would like to ensure that this has a well documented path forward to supporting all ports on a single Listener.

Could make sense to align with NetPol conventions on this (omitting an optional port field implies all), refs kubernetes-sigs/network-policy-api#247 (comment)

[dprotaso]

I was thinking an empty port could be allow the implementation to pick a random port. Though I realize now we could also accomplish that by relaxing our constraints on PortNumber and allowing 0 (unix semantics). If we do that then I don't particularly need an optional port - so I'm curious about other people's use cases.

I think use of port in netpol is fundamentally different than that of a listener - eg. firewall rules semantics (no port == all) vs listener semantics (0 port = wildcard)

[mlavacca]

I think that including L4 considerations at the very first stage of the design, before having the actual API in place is too much of a stretch. Getting rid of L4 routes in favor of including L4 routing information (i.e., backends) is a controversial one (for which I am on the fence honestly) and I don't think that should block this GEP which has been in the work for a long time and needs to be moved forward.

Putting here L4 will likely require a lot of discussion, partially off-topic with the broader goal of this GEP.

[mlavacca]

I'm struggling to understand what this reason means: is it about the Accepted condition of the referenced parent? If that's the case, we are setting here a cross-object dependency. What should we use here if the parent doesn't exist at all (problem raised by the ResolvedRefs condition)?

[dprotaso]

I'm struggling to understand what this reason means: is it about the Accepted condition of the referenced parent?

Yeah - if the ListenerSet tries to attach to a parent Gateway - but that Gateway has Accepted=False

If that's the case, we are setting here a cross-object dependency.

There is a cross object dependency because we expect ListenerSet to have a ParentRef

What should we use here if the parent doesn't exist at all (problem raised by the ResolvedRefs condition)?

hmm... ResolvedRefs condition on *Route seems to only apply to the backend and not parentRefs. Unless we want to make sweeping changes then I'd suggest in your scenario just set Accepted=False with a message that the parent X doesn't exist

[robscott]

I'd recommend copying the Route conditions here since they should be the ~same:

gateway-api/apis/v1/shared_types.go

Lines 317 to 350 in e8ccbe1

	RouteConditionAccepted RouteConditionType = "Accepted"
	// This reason is used with the "Accepted" condition when the Route has been
	// accepted by the Gateway.
	RouteReasonAccepted RouteConditionReason = "Accepted"
	// This reason is used with the "Accepted" condition when the route has not
	// been accepted by a Gateway because the Gateway has no Listener whose
	// allowedRoutes criteria permit the route
	RouteReasonNotAllowedByListeners RouteConditionReason = "NotAllowedByListeners"
	// This reason is used with the "Accepted" condition when the Gateway has no
	// compatible Listeners whose Hostname matches the route
	RouteReasonNoMatchingListenerHostname RouteConditionReason = "NoMatchingListenerHostname"
	// This reason is used with the "Accepted" condition when there are
	// no matching Parents. In the case of Gateways, this can occur when
	// a Route ParentRef specifies a Port and/or SectionName that does not
	// match any Listeners in the Gateway.
	RouteReasonNoMatchingParent RouteConditionReason = "NoMatchingParent"
	// This reason is used with the "Accepted" condition when a value for an Enum
	// is not recognized.
	RouteReasonUnsupportedValue RouteConditionReason = "UnsupportedValue"
	// This reason is used with the "Accepted" when a controller has not yet
	// reconciled the route.
	RouteReasonPending RouteConditionReason = "Pending"
	// This reason is used with the "Accepted" condition when there
	// are incompatible filters present on a route rule (for example if
	// the URLRewrite and RequestRedirect are both present on an HTTPRoute).
	RouteReasonIncompatibleFilters RouteConditionReason = "IncompatibleFilters"
	)

We don't currently require or suggest that implementations populate Route status if the parent is not accepted, and I think that's sensible. If an implementation isn't accepting the parent resource, why would it try to reconcile all of the children of that resource?

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: dprotaso
Once this PR has been reviewed and has the lgtm label, please assign mlavacca for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• geps/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment