Swarm cluster load balancing broken in PhotonOS 4

Describe the bug

Hello

We have a Swarm cluster with multiple managers running on PhotonOS 3 and we are attempting to upgrade it to Photon OS 4. However, in our tests, it seems that the built in load balancing is broken. We see the following behaviour:

Reaching to cluster managers from the outside reaches the service only if the container itself is running on that specific manager node, however if the container is running on a different manager, the curl request is stuck and eventually times out.

We checked the following things:

1. Made sure iptables rules are identical to the working cluster, except for ip and dockers own changes
2. Made sure the sysctl rules are the same: we imported the contents of /etc/sysctl.d and /lib/sysctl.d from the working cluster
3. Tried downgrading docker to the same version as in the working cluster

Unfortunately, no luck so far. Is there any change in PhotonOS 4 that might cause this? We attempted the teste on a brand new cluster.

Reproduction steps

1.Set up a Swarm cluster on PhotonOS 4

Expected behavior

The load balancing should work normally

Additional context

No response

Start by reproducing the reported setup: a Docker Swarm cluster on PhotonOS 4 with the service running on a different manager from the one receiving the request. Review the reported differences in iptables, sysctl settings, and Docker versions against the working PhotonOS 3 cluster. Done means external requests reach the service regardless of which manager hosts its container.