Repository navigation
SecurityGroup: toFQDNs rules never work with Cilium chained on kube-ovn #5131
Description
Describe the bug
toFQDNs egress rules never work when Cilium is chained on kube-ovn, which is the standard Cozystack networking stack. The SecurityGroup API accepts toFQDNs (and projects it 1:1), so a tenant can write a rule that silently never allows anything. Worse, the DNS rule Cilium needs to learn the names (toPorts[].rules.dns) breaks name resolution for the selected pod entirely.
Environment
- Cozystack v1.6.5
- Cilium 1.19.5,
cni-chaining-mode: generic-vethon kube-ovn,kube-proxy-replacement: true - Provider: on-prem
To Reproduce
- In a namespace with no other policy, create a pod and a CiliumNetworkPolicy with an egress rule to kube-dns carrying
rules: {dns: [{matchPattern: "*"}]}on port 53, and atoFQDNs: [{matchName: example.com}]rule on 443. dig example.comfrom the pod times out;curl https://example.comfails.
Expected behaviour
Either the DNS proxy receives the request and toFQDNs works, or the platform documents that FQDN rules are not supported on this stack and the SecurityGroup API rejects them.
Actual behaviour
On the pod's node, cilium-dbg monitor shows the request marked for the proxy (action redirect, -> proxy port <n>), and cilium-dbg status shows the DNS proxy listening with its redirects active. But the proxy never receives the request: no L7 event, empty cilium-dbg fqdn cache list, and the TPROXY rules in CILIUM_PRE_mangle keep a packet count of 0 across queries. With chaining, the host side of the pod's veth is a port of kube-ovn's OVS bridge, so after the BPF program marks the packet it goes into the OVS datapath and never traverses the host IP stack where TPROXY would deliver it to the proxy. Enabling enable-bpf-tproxy on one node (agent restarted with the flag, pod pinned to that node) did not change anything.
Logs
Policy verdict log: ... egress, action redirect, match L3-L4, 10.246.16.194:57902 -> 10.246.6.131:53 udp
-> proxy port 43889 flow 0xc59dd943 , identity 23868->unknown state new ifindex 0 orig-ip 0.0.0.0: 10.246.16.194:57902 -> 10.246.6.131:53 udp
0 0 TPROXY udp -- * * 0.0.0.0/0 0.0.0.0/0 mark match 0x71ab0200 /* cilium: TPROXY to host cilium-dns-egress proxy */ TPROXY redirect 127.0.0.1:43889
Additional context
Any other L7 policy (HTTP rules through Envoy) presumably hits the same path, though only DNS was tested. Until the redirect can reach the host stack, rejecting toFQDNs in pkg/registry/sdn/securitygroup/validate.go, or at least documenting it, would keep tenants from writing rules that cannot work.
Checklist
- I have checked the documentation
- I have searched for similar issues
- I have included all required information
- I have provided clear steps to reproduce
- I have included relevant logs
Activity