Inner Docker Network Connectivity Issues

We are running Ubuntu 24.04 LTS hosts with 1.32 K8s. We've followed the installation manifests and have sysbox-deploy-k8s running on each worker node with no issues/errors. We have made no edits to the mgr or fs settings of the sysbox configmap. K8s uses cilium in full ebpf mode, replacing kube-proxy, if it's relevant.

Inside an ubuntu container using the nestybox docker or docker/systemd images, we cannot seem to get docker networking to work correctly between inner containers.

We've made sure to set the runtimeClass, io.kubernetes.cri-o.userns-mode annotation, and even the container.seccomp.security.alpha.kubernetes.io/dev and container.apparmor.security.beta.kubernetes.io/dev annotations.

Here's a simple example using docker-compose

version: '3.9'

services:
  alpine:
    image: alpine:latest
    command: tail -f /dev/null
    networks:
      - mynet

  nginx:
    image: nginx:latest
    ports:
      - "80:80"
    networks:
      - mynet

networks:
  mynet:

Container-to-container connectivity will work for ICMP (tested with ping) and TCP (tested with netcat), but not using other protocols. If we try to wget or curl the nginx container from the alpine container, we'll get a an error saying server returned error: HTTP/1.1 503 Service Unavailable. This is not, however, an nginx issue, as this error occurs on many different services and types of services. We noticed this because we couldn't get a graphql container to talk to a postgres container while running nhost up.

Networking to the outer container, however, remains unbroken. We can curl nginx fine using localhost:80 from the outer container.

Please let us know what else we can try. I've been trying to get this to work for 2-3 days and I'm losing my mind 😂

Start by reproducing the docker-compose example inside the Ubuntu container using the nestybox Docker or Docker/systemd images, with the listed Kubernetes runtime and security annotations. Compare ICMP, netcat, and curl or wget behavior between the inner containers and the outer container, and inspect sysbox-deploy-k8s, the Sysbox configmap, and the Cilium setup. Done means identifying the cause of non-HTTP protocol connectivity and documenting or fixing the configuration that allows the inner services to communicate.