Favoring "iptables-legacy" on Debian Sid causes iptables rules inconsistency

Description of Issue/Question

Debian Sid (Unstable) uses iptables-nft by default now to provide /sbin/iptables and still provides a /usr/sbin/iptables-legacy for compatibility.
Recent change in docker package that was released into Debian Sid on 1/17/2019 forces docker to favor iptables-legacy binary over the "default" iptables.
This causes "normal" iptables rules to show up under iptables -S and docker’s rules only under iptables-legacy -S

Specific impact in my setup was inconsistent iptables configuration and the docker container could not initiate outgoing connections any more.

Setup

I recently upgraded docker package in Debian:

• from: docker.io 18.06.1+dfsg1-2
• to docker.io 18.06.1+dfsg1-3

Which on the Debian side has the following change log:
https://tracker.debian.org/news/1021701/accepted-dockerio-18061dfsg1-3-source-all-amd64-into-unstable/

Which seems to introduce the following docker commit:
https://github.com/docker/libnetwork/commit/7da66eea9f68e4abc83ed2892114ec565eddd66a#diff-b90cadcd0928c1e490272f4761a52bac

Checking in the Debian source repo confirms the suspicion:
docker.io 18.06.1+dfsg1-2 https://sources.debian.org/src/docker.io/18.06.1+dfsg1-2/libnetwork/iptables/iptables.go/

• does not contain the commit

docker.io 18.06.1+dfsg1-3 https://sources.debian.org/src/docker.io/18.06.1+dfsg1-3/libnetwork/iptables/iptables.go/

• contains the commit in line 90

Workaround

When forcing the system to default to iptables-legacy, all rules will show up under iptables-legacy. In my case, the container was then able to initiate outgoing connections again

update-alternatives --set iptables /usr/sbin/iptables-legacy

Logs and other Evidence

18.06.1+dfsg1-2:

• Iptables-nft is the default
• All rules show up under iptables -S (observe custom rule -A INPUT -s 15.15.15.51 -j DROP)

root@debian:/home/test# dpkg -l |grep docker.io
ii  docker.io                        18.06.1+dfsg1-2              amd64        Linux container runtime
ii  python3-docker                   3.4.1-4                      all          Python 3 wrapper to access docker.io's control socket
root@debian:/home/test# /sbin/iptables -A INPUT -s 15.15.15.51 -j DROP
root@debian:/home/test# /sbin/iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-USER
-N DOCKER-ISOLATION-STAGE-2
-A INPUT -s 15.15.15.51/32 -j DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8200 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-USER -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
root@debian:/home/test# /usr/sbin/iptables-legacy -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
root@debian:/home/test# update-alternatives --display iptables
iptables - auto mode
  link best version is /usr/sbin/iptables-nft
  link currently points to /usr/sbin/iptables-nft
  link iptables is /usr/sbin/iptables
  slave iptables-restore is /usr/sbin/iptables-restore
  slave iptables-save is /usr/sbin/iptables-save
/usr/sbin/iptables-legacy - priority 10
  slave iptables-restore: /usr/sbin/iptables-legacy-restore
  slave iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 20
  slave iptables-restore: /usr/sbin/iptables-nft-restore
  slave iptables-save: /usr/sbin/iptables-nft-save
root@debian:/home/test#

18.06.1+dfsg1-3:

• Iptables-nft is the default
• Custom rules show up only under iptables -S (observe custom rule -A INPUT -s 15.15.15.51 -j DROP)
• Docker rules show up only under iptables-legacy -S

root@debian:/home/test# dpkg -l |grep docker.io
ii  docker.io                        18.06.1+dfsg1-3              amd64        Linux container runtime
ii  python3-docker                   3.4.1-4                      all          Python 3 wrapper to access docker.io's control socket
root@debian:/home/test# /sbin/iptables -A INPUT -s 15.15.15.51 -j DROP
root@debian:/home/test# /sbin/iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -s 15.15.15.51/32 -j DROP
# Warning: iptables-legacy tables present, use iptables-legacy to see them
root@debian:/home/test# /usr/sbin/iptables-legacy -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8200 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
root@debian:/home/test# update-alternatives --display iptables
iptables - auto mode
  link best version is /usr/sbin/iptables-nft
  link currently points to /usr/sbin/iptables-nft
  link iptables is /usr/sbin/iptables
  slave iptables-restore is /usr/sbin/iptables-restore
  slave iptables-save is /usr/sbin/iptables-save
/usr/sbin/iptables-legacy - priority 10
  slave iptables-restore: /usr/sbin/iptables-legacy-restore
  slave iptables-save: /usr/sbin/iptables-legacy-save
/usr/sbin/iptables-nft - priority 20
  slave iptables-restore: /usr/sbin/iptables-nft-restore
  slave iptables-save: /usr/sbin/iptables-nft-save
root@debian:/home/test#

Start in libnetwork/iptables/iptables.go, especially the change around line 90 referenced in the issue, and compare it with the Debian Sid iptables-nft and iptables-legacy setup described here. Reproduce the split-rule behavior and determine the expected backend selection; done means Docker rules and normal rules use the same iptables implementation and container networking works.