How can I configure an extra cluster that I need in my (Http)Filter

[aukevanleeuwen]

Hello,

I'm using a Lua filter to send off some information to an external HTTP service (using https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/lua_filter#httpcall). For this I need an extra backend to exist. Currently I've used an EnvoyPatchPolicy to do this, but as soon as I get into - for example - upstream TLS configuration it becomes difficult to manage. I've tried to add another 'orphaned' Backend, but that doesn't result in an Envoy cluster automagically. Probably because it's orphaned, there are no HttpRoutes connected to it. It would be nice if that would work however, because that way I should also be able to add a BackendTLSPolicy to it.

I thought of using a 'dummy' listener and try to hide that from being exposed as a LoadBalancer type service. Or perhaps adding another HttpRoute to the 'main' listener that will always result in a negative match (i.e. adding a header match on something like X-Foo == true && X-Foo != true), but it seems rather silly.

Are there other ways that I'm overlooking?

[zhaohuabing]

@aukevanleeuwen as a workaround, creating a dummy Listener or HttpRoute would work.
Think it loud—should we consider adding direct support for this within the Lua extension? cc @envoyproxy/gateway-maintainers @rudrakhp

The following YAML snippet is a rough idea how it might look within the lua extension:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end
      backendRefs:
        - name: cluster-used-by-lua
          port: 8080  

[aukevanleeuwen]

Without knowing exactly how the code currently works, it seems as if it's first constructing a 'graph' to figure out if Backends should be created or not. I.e. if the backend isn't referenced 'via' a listener → route, it's not considered.

My question would be: would it hurt to just create them either way? Or doesn't a Backend necessarily to a Cluster? Probably there is some naming here and there (on the Backend) that I see that seems to include the listener/httproute information, but maybe that can default to just the backend name?

This is just me "not hindered by any knowledge" (as we say in Dutch) just asking the question.

[aukevanleeuwen]

Btw: I'm digging though the listener / http-route apis to figure out the best way to do this. Do you have a suggestion for the dummy route / listener?

I see two three two ways:

1. Add another route rule to the existing HttpRoute. However, since my main rule essentially covers all traffic (path: '/'), this rule would be more specific and if I read it correctly would always be evaluated earlier. Since this rule would have some logic of always not matching, it's a waste though.
2. Add both a listener and a route. The upside is that it wouldn't have this redundant matching like (1), a Listener will create a service of type LoadBalancer which triggers all sorts of automatically exposing this etc. I'd rather just have that listener be a ClusterIP perhaps? Is that easily doable?
3. I was reading the HttpRoute rules again and I see that you can reference multiple backends there. If I would add another backend to the 'main' route with weight: 0 would that work and would it be a good solution?

Thanks for your answer(s)!

[arkodg]

the extra cluster can be created using EnvoyPatchPolicy e.g. from the inference extension project https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/release-v0.1.0/pkg/manifests/gateway/patch_policy.yaml

[rudrakhp]

IMO backends don't belong under Lua EEPs, speaking strictly in terms of usecase the API is supposed to cater to. As suggested by @arkodg can Lua EEP + Patch policy be used together for this requirement?

[aukevanleeuwen]

@rudrakhp / @arkodg

the extra cluster can be created using EnvoyPatchPolicy e.g. from the inference extension project https://github.com/kubernetes-sigs/gateway-api-inference-extension/blob/release-v0.1.0/pkg/manifests/gateway/patch_policy.yaml

Indeed it's possible (as I said in the original question I've been doing it this way at first) but it becomes rather hairy if you want to - in my case - add specific (m)TLS configuration to it. So on a Backend I can use a BackendTLSPolicy with a caCertificateRefs for certificate pinning. But if I have to do that with EnvoyPatchPolicy it would be difficult I suppose? I'd probably have to mount a configmap via spec.provider.kubernetes.envoyDeployment.container in EnvoyProxy (?) and then reference that file in the patch?

I haven't tested it yet, but I'm kind of assuming that the BackendTLSPolicy would actually be able to rotate those trusted certificates without having to a restart the deployment? That wouldn't work with patching I suppose right? Because the configmap is just mounted and a change to the configmap is not picked up until the pod is restarted (AFAIK).

Now that I'm thinking about it, even if I would use the workaround with the 'dummy' listener / http-route, I'd still be in trouble with the client certificate (mTLS). Because currently that's implemented at spec.backendTLS in EnvoyProxy and obviously is not aware of a secondary backend? Isn't that odd from an API design perspective? You can have multiple routes with different backends, but there is a single EnvoyProxy configuration for 'backend TLS'?

I'm all ears for better options. I have no issues with using envoy patch policy, but it would be really nice if there was a way in which some of these secrets (client certificate, TLS truststore/ca) could be provided to the envoy configuration without having to deal with an immutable configmap mount. Any help is appreciated!