Race condition between netpol and IPVS based ipset updates #1732
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
{{ message }}
What happened?
I am observing a race condition between the
NetworkPolicyControllerand theNetworkServicesControllerwhen updating IPVS entries. The scenario is as follow:kube-routerruns the periodicsyncIpvsFirewalland adds the ExternalIP to thekube-router-svip-prtipset. Here traffic to the ExternalIP coming from other nodes start being ACCEPT-ed byiptables. At this stage,NetworkServicesControlleralso adds the ExternalIP to theipSetHandlersmap it maintains in memory.kube-routerrunssyncNetworkPolicyChains. This refreshes ipsets to include IPs contained in NetworkPolicies, starting from the in memory values that theNetworkPolicyControllerholds in itsipSetHandlers.NetworkPolicyControlleripSetHandlersmap doesn't know anything about the ExternalIP that was added by theNetworkServicesController, and hence it is removed fromkube-router-svip-prt. Traffic to the ExternalIP gets REJECT-ed by itlables, untilsyncIpvsFirewallruns again.What did you expect to happen?
The ExternalIPs of services should be added to the
kube-router-svip-prtipset and remain there, instead of getting removed and re-added.How can we reproduce the behavior you experienced?
Steps to reproduce the behavior:
a.b.c.d.kube-router-svip-prtipset on the host where the pod started withipset list kube-router-svip-prt | grep -P "a\.b\.c\.d"kube-routerrunssyncIpvsFirewalland will disappear whenkube-routerrunsfullPolicySync.System Information (please complete the following information)
Kube-Router Version (
kube-router --version):Running kube-router version v2.1.0-11-gac6b898c, built on 2024-03-18T20:39:38+0100, go1.22.0Kube-Router Parameters:
kubectl version) : 1.27.13Logs, other output, metrics
This is what i see in logs (I extracted the relevant parts, the full run is attached)
When ipsets are restored by the
NetworkServicesControllerthekube-router-svip-prtcontains87.250.179.246, while when they are restored by theNetworkPolicyController87.250.179.246is missing.I am patching the issue for now by running
ipset.Save()at each controller before they build their updated version, to make sure the base layer is the current config, instead of the previous inmemory content which might be outdated.kube-router-ipset-race.log
The text was updated successfully, but these errors were encountered: