[RFC] Custom port-mappers

Description

This is a proposal for adding support for custom port-mappers to the Engine through a new type of plugins. This could be useful in a number of case:

• For Engine integrators dealing with two “hosts” (e.g. Docker Desktop, with a host and a guest; or rootlesskit, with a host and a separate netns), where HostIP and HostPort aren’t designating the Engine host.
• To let us (or the community) experiment with new port-mappers (e.g. bpfilter)
• Map ports through vendor-specific APIs (e.g. cloudflared)
• Decouple the userland-proxy from the ‘nat’ port-mapper, such that the userland-proxy can be used standalone. This would give a way for ufw users to map ports without bypassing ufw rules (https://github.com/moby/moby/issues/22054).
  • We could even disable the ‘nat’ port-mapper and keep only the ‘userland-proxy’ mapper whenever ufw is detected.

The current port-mapping code would be broken down into a few builtin mappers.

Gw modes are partially intersecting with port-mappers as they both define how ports should be mapped. To make it easy to use, the default port-mapper will be picked based on the network’s gw_mode.

• nat: default port-mapper for networks gw_mode=nat
• routed: default port-mapper for networks gw_mode=routed
• rootless: default port-mapper when the daemon is running in rootless mode
• userland-proxy: default port-mapper when ufw is in use? Unavailable when the userland-proxy has been disabled.
• remote: adapter for remote plugins. This gives us the ability to change the internal API without affecting the plugin API.

Tentative schedule

• Experimental support in v29 (RC1 scheduled for July 10).
  • The PortMapper API described below is fully implemented. The daemon receives a new feature flag and config option to define which port mappers are enabled.
  • Default portmapper is determined based on network’s config.
• Probably in a v29 minor (if they can’t make it before we cut RC1):
  • Add the ability to specify a per-mapping port-mapper to the Engine API / CLI / Compose.
  • PortMapper API is subject to change based on feedback, issues discovered, etc…
  • nftables support
• GA in v30.0

Open questions

• Should custom port-mappers write to the same iptables chains as the Engine? (e.g. the desktop port-mapper would need to add a rule in filter-FORWARD / filter-DOCKER)
  • This is not really a problem for nftables -- custom mappers can define their own custom chain, and use the appropriate priority to slip in at the right place.
  • For iptables, since it's going to be deprecated, and we don't expect custom mappers to need more that inserting rules in filter-FORWARD, it's fine to reuse existing Engine chains.

UX

Daemon:

• [v29.0] Enable support for port-mapper plugins through a new feature flag port-mapper-plugins.
• [v29.0] A new key default-port-mappers is added to the daemon config.

CLI / Compose:

• [v29.x] CLI: Change the -p microformat to add the ability to specify a port-mapper: -p spec=127.0.0.1:8080:80/tcp,mapper=cloudflared.
• [v29.x] Compose: Change the ports property to add a new mapper subproperty to the long syntax.

Plugin API

Port-mappers are either builtin, or Docker plugins (v1 or v2). Plugins’ /Plugin.Activate endpoint return PortMapper.

Port-mapper API

The PortMapper API defined here is based on the existing code:

• PortBindingReq is based on portBindingReq (see here).
• PortBinding is based on portBinding (see here)

type Protocol uint8 // Protocol is either syscall.IPPROTO_TCP (= 6), syscall.IPPROTO_UDP (= 17) or syscall.IPPROTO_SCTP (= 132).

// PortBindingReq is sent to the PortMapper to request a port mapping.
type PortBindingReq struct {
	Proto       Protocol
	BackendIP   netip.Addr
	BackendPort uint16
	// FrontendIPs are group of IP addresses that should get the same port assigned
	// when an ephemeral port, or a port range is specified in FrontendPort and
	// FrontendPortEnd. Each FrontendIP is associated to a request ID, and should
	// yield an independent PortBinding.
	FrontendIPs     []netip.Addr
	// FrontendPort is either an exact port, an ephemeral port (= 0), or the start
	// of a port range.
	FrontendPort uint16
	// FrontenPortEnd should be the same as HostPort when an exact or ephemeral
	// port is requested. Otherwise it should be the end of the port range.
	FrontendPortEnd uint16
	// Labels set on the container. PortMappers might use it to define extra
	// parameters.
	Labels map[string]string
}

// PortBinding is returned by the PortMapper when a request has been processed.
type PortBinding struct {
	FrontendIP      netip.Addr

	// FrontendPort is the frontend port picked by the PortMapper when an ephemeral port, or a port range was specified in the request.
	FrontendPort uint16
}

type PortBindingResp struct {
	PortBinding

	// Sockfd is a listenable socket fd (i.e. socket(2) + bind(2)) received
	// through an ancilliary message (or in-process, for builtin mappers).
	// It'll be used by the userland-proxy if it's enabled. If no userland-proxy
	// is required, this must be nil.
	Sockfd *os.File

	// DiscardReq is returned when the PortBindingReq can't be fulfilled, but the
	// driver deems it's not a fatal error.
	DiscardReq bool

	// Error indicates the reason why the request could not be fulfilled. If
	// DiscardReq is false (fatal error), the error is displayed to the
	// end-user. Otherwise, it's logged and discarded.
	//
	// Common errors are: 'no free port in range', 'could not find an ephemeral
	// port'.
	Error *string
}

type PortMapper interface {
	// MapPorts takes a list of port binding requests, and returns a map of 
	// request IDs associated to PortBindings. Both lists MUST have the same size.
	//
	// Multiple port bindings are passed when they're all requesting the
	// same port range, or an ephemeral port, over multiple IP addresses and
	// all pointing to the same container port. In that case, the PortMapper
	// MUST assign the same HostPort for all IP addresses.
	//
	// When an ephemeral port, or a single port from a range is requested
	// MapPorts should attempt a few times to find a free port available 
	// across all IP addresses.
	MapPorts(addrs []PortBindingReq) []PortBindingResp
	
	// UnmapPorts takes a list of port bindings to unmap.
	UnmapPorts(addrs []PortBinding)
}

Implementations outline

All port-mappers have a set of common steps executed before calling a port-mapper: 1. set the default HostIP (if available); 2. build a list of related portBindingReq (i.e. all reqs should be allocated the same port).

After the port-mapper returns, the userland-proxy may be started, if it’s enabled, and Sockfd is non-nil. Custom plugins that don’t want / need the userland-proxy should return a nil Sockfd.

The firewaller.Network.AddPorts method is split into two methods: one to set up DNAT / MASQUERADE rules (used by the nat and rootless port-mappers); another to add a rule in filter-FORWARD to accept NATed / routed packets on the container’s published ports (used by nat, routed, rootless port-mappers).

A new method is added to firewaller.Network to insert filter-ACCEPT rules (used by userland-proxy).

PortAllocator.RequestPortsInRange will be updated to make it responsible for allocating OS ports. The same PortAllocator will be used by all the builtin port-mappers that need it (i.e. nat, rootless, userland-proxy).

Here's an outline of what each port-mapper would do, including all builtin mappers, and an (hypothetical) custom mapper for Cloudflare Tunnels.