External networks attached to services consistently leak IP addresses.

Description

When using docker swarm, there are situations when attempting to remove a stack will cause IP addresses on an external overlay network to leak, consistently.
The outcome of this problem is that the subnet assigned to the network will have all its IPs eventually depleted.
You can observe this with docker network inspect {network} in the [0].Status.IPAM.Subnets.{subnet}.IPsInUse field which increments by one after a deploy and subsequent removal.

Expected Behavior

We should expect after deploying a stack and removing that stack that the IPs get reclaimed and the IPsInUse field remains the same before and after.

Steps to Reproduce

• Initialize a swarm node and create an external overlay network.
• Create a compose file which uses both that external overlay network and an internal network in a service.
• Check the IPsInUse field.
• Deploy the stack.
• Remove the stack.
• The IPsInUse field should now be one higher.

Considerations

Let's call the service which has both the external overlay network and the internal network app
This problem only occurs when the internal network is removed prior to the container of the app service being fully removed.
Thus, you can perform docker service rm app then docker stack rm and the problem disappears.
Alternatively, you can recreate the problem with docker service rm app and before that container is removed, run docker network rm {internal_network}. This mimics docker stack rm behavior, which runs those commands sequentially asynchronously.

Another observation is that after restarting docker, the IPsInUse field gets reset back to an expected value. It does seem as if these IPs are truly freed because I can deploy services successfully even if the subnet was depleted before the reset.
Similar to this observation is that on some configurations, docker network inspect {network_name} and docker network inspect {network_id} differ, the {network_name} inspect showing the leaked IPs but the {network_id} showing the expected IP allocations. This doesn't happen on every machine, however.

Configuration

Docker version 29.2.1, build a5c7197

Start by reproducing the swarm deployment and removal sequence with an external overlay network, an internal network, and a service attached to both; inspect the network's [0].Status.IPAM.Subnets.{subnet}.IPsInUse before and after. Trace the asynchronous ordering between internal-network removal and complete service-container removal. Done means repeated stack deploy/removal reclaims the address and name- and ID-based network inspection agree.