4.23.0.0 VR: PREROUTING-ethX-def saves CONNMARK before setting MARK, static NAT replies leave via the wrong interface #14464

problem

Isolated network router, source NAT on eth2 (192.0.2.43), static NAT IP 203.0.113.114 on eth3
(second VLAN) to VM 10.168.10.37. After Disable/Enable Static NAT the mangle chain was:

-A PREROUTING -s 10.168.10.37/32 -m state --state NEW -j PREROUTING-eth3-def
-A PREROUTING-eth3-def -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING-eth3-def -j MARK --set-xmark 0x67/0xffffffff

CONNMARK --save-mark runs before MARK, so the connection is stored with mark=0. The first packet
(SYN) is routed by fwmark 0x67 to Table_eth3 and SNATed to 203.0.113.114, but later packets of the
connection get mark 0 from --restore-mark, follow the main table out eth2 with source
203.0.113.114 and are dropped upstream. "curl -4 ifconfig.me" in the VM hangs; conntrack shows
ESTABLISHED/UNREPLIED entries with mark=0. On an earlier router build the order was MARK then
CONNMARK --save-mark and it worked.

versions

CloudStack 4.23.0.0, KVM, systemvm template 4.22.0 (router reports version 4.22.0), isolated network.

The steps to reproduce the bug

1. Isolated network with source NAT on the untagged range; acquire a public IP from a second VLAN.
2. Enable Static NAT to a VM, then Disable and Enable it again.
3. In the router: iptables -t mangle -S PREROUTING-ethX-def -> CONNMARK --save-mark comes first.
4. In the VM: curl -4 ifconfig.me hangs; conntrack -L -s shows mark=0.

What to do about it?

Generate the chain with MARK before CONNMARK --save-mark (append in that order, or insert both
at fixed positions). Workaround: iptables -t mangle -I PREROUTING-ethX-def 1 -j MARK --set-xmark 0x6N
and conntrack -D -s .