Epic: real-world experience migrating from docker to nerdctl

I am using a home-grown deployment tooling (similar to terraform-docker) that heavily relies on a (docker) cli to bring up networks, volumes and containers to a desired state on remote iot nodes.

The tool connects over ssh to the node, evaluate the current state of the deployment then delete / create the necessary resources to bring it to the desired state.

I first tried to migrate from docker last year - quickly faced with a large number of failures - this is when I got involved with nerdctl to fix them.

Trying again a year after, things are better, but there are still a number of blocking issues.

This ticket is meant to keep track of all of them. Eventually, they can be broken down in individual tickets to be fixed when they have a simple reproducer or a diagnosis (some of them already are).

**Currently blocking:**
- [ ] containerd/nerdctl#3663
- [ ] containerd/nerdctl#3668 (while technically not a part of the deployment, this one is also hitting my lifecycle)

**Problematic, but not blocking:**
- [ ] containerd/nerdctl#4283
- [x] containerd/nerdctl#4157

**Blocking using rootless:**
- [ ] containerd/nerdctl#4276
- [ ] it is not clear that `dhcp` would work rootless

**Blocking using DHCP:**
- [ ] cni dhcp plugin does not seem to work well when there are multiple interfaces - in my context, `eth0` is not connected, but `wlan0` is - yet, `dhcp` insists on trying to bring up `eth0` and get a lease of it, seemingly ignoring `wlan0` - of course, it deadlines trying to do so
- [ ] cni dhcp plugin will block `nerdctl run`: in failing above, nerdctl run will get stuck (waiting for dhcp), with no timeout - note that the above is in the context of a vlan network which parent is `wlan0` - either I do not know how to use the dhcp plugin properly, or this thing is just not ready

**Minor / can be workaround / can live without for now:**
- [ ] if `uidmap` is missing on the system, rootlesskit error is confusing / unhelpful - rootlesskit should test for the presence of the required binaries and provide a helpful message instead
- [x] containerd/nerdctl#4282

**Fixed / patch pending:**
- [ ] [PRIVATE PATCH] containerd/nerdctl#4281
- [x] somehow, nerdctl has a propensity to create broken containers (in `unknown` state), putting the system in a corrupted state that cannot be recovered from using nerdctl commands - and _maybe_ show underlying containerd issues - this is being discussed in containerd/containerd#11890 (pending upstream fix, thanks to @ningmingxiao)

There might very well be more once I get past these. Will update this ticket accordingly.