Cillium, Gateway, Loadbalancer issues.

[tradenet]

I have a Talos k8s cluster with Cilium. Set up a LoadBalancer pool with Cilium.
I have deployed/install NGINX Gateway Fabric directly from the OCI helm registry.
It appears that the LoadBalancer ip lease is granted to the Nginx Fabric deployment by Cilium.
Deployed the coffee example.
However, the LoadBalancer ip isn't reachable. There seems to be a conflict with the coffee gateway deployment.
However, I can reach the coffee example at the node ip address and designated port.
I am not sure what I am missing here.

 k get svc,pods
 NAME                               TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
 service/coffee                     ClusterIP      10.96.15.114    <none>           80/TCP                       16m
 service/ngf-nginx-gateway-fabric   LoadBalancer   10.96.235.200   192.168.60.131   80:31991/TCP,443:31418/TCP   45h
 service/tea                        ClusterIP      10.96.232.85    <none>           80/TCP                       16m
 
 NAME                                            READY   STATUS    RESTARTS   AGE
 pod/coffee-676c9f8944-88jsw                     1/1     Running   0          16m
 pod/ngf-nginx-gateway-fabric-69f9556b54-rg52z   2/2     Running   0          45h
 pod/tea-6fbfdcb95d-t5f64                        1/1     Running   0          16m`
 

> Gateway:
> `Name:         gateway
> Namespace:    nginx-gateway
> Labels:       <none>
> Annotations:  <none>
> API Version:  gateway.networking.k8s.io/v1
> Kind:         Gateway
> Metadata:
>   Creation Timestamp:  2025-04-02T13:58:33Z
>   Generation:          1
>   Resource Version:    20504062
>   UID:                 ab4868fc-c776-4ad8-960f-3f47690899f6
> Spec:
>   Gateway Class Name:  nginx
>   Listeners:
>     Allowed Routes:
>       Namespaces:
>         From:  Same
>     Hostname:  *.talos.local
>     Name:      http
>     Port:      80
>     Protocol:  HTTP
> Status:
>   Addresses:
>     Type:   IPAddress
>     Value:  192.168.60.131
>   Conditions:
>     Last Transition Time:  2025-04-02T13:58:33Z
>     Message:               Gateway is accepted
>     Observed Generation:   1
>     Reason:                Accepted
>     Status:                True
>     Type:                  Accepted
>     Last Transition Time:  2025-04-02T13:58:33Z
>     Message:               Gateway is programmed
>     Observed Generation:   1
>     Reason:                Programmed
>     Status:                True
>     Type:                  Programmed
>   Listeners:
>     Attached Routes:  2
>     Conditions:
>       Last Transition Time:  2025-04-02T13:58:33Z
>       Message:               Listener is accepted
>       Observed Generation:   1
>       Reason:                Accepted
>       Status:                True
>       Type:                  Accepted
>       Last Transition Time:  2025-04-02T13:58:33Z
>       Message:               Listener is programmed
>       Observed Generation:   1
>       Reason:                Programmed
>       Status:                True
>       Type:                  Programmed
>       Last Transition Time:  2025-04-02T13:58:33Z
>       Message:               All references are resolved
>       Observed Generation:   1
>       Reason:                ResolvedRefs
>       Status:                True
>       Type:                  ResolvedRefs
>       Last Transition Time:  2025-04-02T13:58:33Z
>       Message:               No conflicts
>       Observed Generation:   1
>       Reason:                NoConflicts
>       Status:                False
>       Type:                  Conflicted
>     Name:                    http
>     Supported Kinds:
>       Group:  gateway.networking.k8s.io
>       Kind:   HTTPRoute
>       Group:  gateway.networking.k8s.io
>       Kind:   GRPCRoute
> Events:       <none>

Docs show that if there is a conflict, it reverts to the pod ip. Which is the case here.
nmap -Pn -n -p 31991 192.168.60.0/24 -vv | grep 'Discovered' Discovered open port 31991/tcp on 192.168.60.33
Any ideas? Doing a port scan doesn't reveal the service/ngf-nginx-gateway-fabric LoadBalancer to be reachable, with the coffee example or without.
Not sure what I am missing here.
Thanks.

[sjberman]

@tradenet What is the response you are getting when attempting to send traffic to the ngf-nginx-gateway-fabric service? If you port-forward that service to your localhost, can you send traffic properly? The load balancer IP appears to be allocated, but this is also at the infrastructure level and above the scope of NGINX Gateway Fabric.

(sidenote: I recommend deploying your apps and gateway objects in a separate namespace from nginx-gateway. That namespace is meant to be dedicated to NGINX Gateway Fabric, though this probably isn't the cause of the traffic problems)