Adds EP-10411: Gateway API Inference Extension Support #10420
Conversation
danehans
added
Type: Docs
Area: Gloo Plugin
Area: Extensibility
Area: K8S Gateway API
|
cc: @EItanya |
Signed-off-by: Daneyon Hansen <[email protected]> Co-authored-by: changelog-bot <changelog-bot> Co-authored-by: Daneyon Hansen <[email protected]> Co-authored-by: Yuval Kohavi <[email protected]> Co-authored-by: Sam Heilbron <[email protected]> Co-authored-by: Nathan Fudenberg <[email protected]> Co-authored-by: Ryan Old <[email protected]> Co-authored-by: soloio-bulldozer[bot] <48420018+soloio-bulldozer[bot]@users.noreply.github.com> Co-authored-by: Bernie Birnbaum <[email protected]> Co-authored-by: Rachael Graham <[email protected]> Co-authored-by: Ravinder Punia <[email protected]> Co-authored-by: sheidkamp <[email protected]> Co-authored-by: Jenny Shu <[email protected]> Co-authored-by: Nadine Spies <[email protected]> Co-authored-by: David Jumani <[email protected]> Co-authored-by: Eitan Yarmush <[email protected]> Co-authored-by: Nina Polshakova <[email protected]> Co-authored-by: Steven Landow <[email protected]> Co-authored-by: Ariana W. <[email protected]>
docs/enhancements/10411.md
Outdated
| ### Configuration | ||
|
|
||
| * Update the [configuration](https://github.com/k8sgateway/k8sgateway/blob/main/install/helm/gloo/generate/values.go) API to enable/disable this feature. | ||
| * Update Helm charts to install/uninstall k8sgateway with this feature based on user-provided configuration. |
There was a problem hiding this comment.
If this is enabled via GatewayParameters, does it require helm charts change? Trying to understand if this is configured at install time of CP or deployment of the gateway DP?
There was a problem hiding this comment.
A user should be able to set a Helm knob to enable this feature. The internal config plumbing should convert the helm know into the equivalent GatewayParameters setting. This will avoid users having to configure the GatewayParameters for GIE during runtime. Depending on the time required to implement this EP, I may defer the Helm piece as follow-up work.
There was a problem hiding this comment.
SGTM thanks for the explanation!
Adds Enhancement Proposal 10411 that proposes adding Gateway API Inference Extension support. Signed-off-by: Daneyon Hansen <[email protected]>
Signed-off-by: Daneyon Hansen <[email protected]>
Signed-off-by: Daneyon Hansen <[email protected]>
|
@danehans Just a heads up that we moved the enhancements repository outside of the docs/ directory into https://github.com/kgateway-dev/kgateway/tree/main/design. |
There was a problem hiding this comment.
The enhancements template was moved to design: https://github.com/kgateway-dev/kgateway/tree/main/design.
|
|
||
| ### Configuration | ||
|
|
||
| * Update the [configuration](https://github.com/kgateway-dev/kgateway/blob/main/install/helm/gloo/generate/values.go) API to enable/disable this feature. |
There was a problem hiding this comment.
We ripped out the legacy chart (and therefore the values.go generate file). See https://github.com/kgateway-dev/kgateway/tree/main/install/helm/kgateway for the new location.
| * Secure the gRPC connection between Gateway and GIE implementations. | ||
| * Support kgateway upgrades when this feature is enabled. | ||
|
|
||
| ## Implementation Details |
There was a problem hiding this comment.
We can update this proposal as we iterate on the initial implementation, but I think it would be nice to sketch out the helm chart / API changes, e.g. GatewayParameter, changes for historical context.
|
|
||
| ### Plugin | ||
|
|
||
| * Add GIE as a supported [plugin](https://github.com/kgateway-dev/kgateway/tree/main/projects/gateway2/extensions2/plugins). The plugin will manage [Endpoints](https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/endpoint/endpoint.html) based on the InferencePool resource specification. The Gateway implementation, e.g. Envoy proxy, will forward matching requests using the [External Processing Filter](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/ext_proc/v3/ext_proc.proto#external-processing-filter-proto) to the ES deployment. The ES is responsible for processing the request, selecting an Endpoint, and returning the selected Endpoint to Envoy for routing. |
There was a problem hiding this comment.
Move the EP from docs/enhancements/10411.md to design: https://github.com/kgateway-dev/kgateway/tree/main/design
Adds Enhancement Proposal 10411 that proposes adding Gateway API Inference Extension support.
Supports #10411