Floating IPs should optionally attach to a specific interface

At present, when a floating IP address is attached to an instance, traffic for that floating IP address is directed to the primary interface of the guest. This makes it difficult or impossible to expose a service to the outside world using the same port as a different service that is consumed within the VPC.

For example, consider a GitHub-like service that offers a custom SSH endpoint on port 22. End users, when connecting to the service, should reach this custom SSH service. The operator of the application still needs to be able to use the regular system SSH service on port 22 to manage the instance.

Imagine that an instance has a primary interface with IP address 10.0.0.1. The system SSH service binds, by default, to 0.0.0.0:22, and will thus accept connections on any interface, including the primary. If we create a second interface, with IP address 10.0.0.2, then we can use a more specific bind address for the custom SSH service: 10.0.0.2:22.

When we attach the floating IP to the instance, we need to be able to tell the system to direct the traffic for that floating IP to a specific interface: the second interface where the application is listening for outside traffic. That way, operators and automation can continue to access the guest via SSH using the primary IP address. When end users connect to port 22 on the floating IP, which listed in DNS as the public name for the service, they are directed to the custom application instead of the system SSH server.

A related challenge emerges when an application needs to be accessible via several floating IP addresses, but needs to be able to tell which floating IP was in use by the client when it connected. Today, that does not appear to be possible: as part of translating the connection to the floating IP, the destination address (previously, the floating IP) is rewritten to use the address of the primary interface, so as to be legible to the guest operating system. If you attach a second floating IP to an instance, it would also be directed to the primary IP. If we could nominate a specific network interface when attaching an external IP address, then the operator could add a series of secondary interfaces to an instance, and direct each floating IP to a unique guest-visible destination address.

Note that AWS has this as a first class concept with their Elastic IP abstraction: when associating an Elastic IP address with an interface, you choose a particular network interface, rather than just an instance.

The issue names no files, tests, or entry points. Start by tracing the floating-IP attachment and destination-translation paths; done means an attachment can select a guest interface and preserve the selected guest-visible destination for multiple floating IPs.