cloud-init init stage crashes with IsADirectoryError when /var/lib/cloud/instance is a directory instead of a symlink

# Bug report

`cloud-init`'s `init` stage (run by `cloud-init.service`, after
`cloud-init-local.service`/`init-local` has already completed
successfully) intermittently aborts with:

```
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'
```

`/var/lib/cloud/instance` is expected to always be either a symlink into
`/var/lib/cloud/instances//`, or absent. When it is found to be a
real (non-symlink) directory instead, `Init._reflect_cur_instance()`'s
`util.del_file(self.paths.instance_link)` call — a plain `os.unlink()` —
raises this exception uncaught, and the whole `init` stage aborts before
any `cloud-config` modules (`modules-init`/`modules-config`/
`modules-final`) run. Downstream effects observed: any `bootcmd`/`runcmd`
configured in user-data never executes, and anything polling
`cloud-init status` for the terminal state never sees `succeeded` — it
sees `failed` instead, with the above message as the recorded `init`
stage error.

## Why I don't think this is "just" a duplicate of #4282 / #3710

- **#3710 / LP:#1883903** (2020) diagnosed one specific cause
(`cc_final_message` promoting the path via an unguarded
`write_file(..., ensure_dir_exists=True)`) and was fixed in 20.3.
- **#4282** (2023, cloud-init 23.1.2) reported the identical symptom
recurring on a newer version, but was closed `not_planned` in 2024 for
lack of a reproduction, with no root cause identified for that
recurrence.
- This report is on cloud-init **22.3.4** (between the two above), with a
more detailed reproduction shape that neither prior report had (see
below), which may help move this forward: it is **reproducible at a
roughly consistent, low rate** (not a one-off), it is **VMM-independent**
(ruling out a hypervisor-specific cause), and there is a structural
observation about *when in the boot* the corruption must occur that I
don't believe either prior report established.

## Steps to reproduce the problem

I don't yet have a minimal, deterministic repro (see "what I've ruled out"
below) — this is from a large-scale, repeated-boot test environment, not a
hand-crafted reproduction. Reporting the observed pattern in case it helps
someone else reproduce it or spot the mechanism:

- Boot many short-lived VMs in a loop from the **same shared, read-only,
offline-prepared base image** (each VM gets its own writable overlay on
top of that same base image; the base image itself is never written to
by any individual VM boot).
- Use a `NoCloud`-style user-data seed that configures several `bootcmd`
entries (mounting a handful of attached block devices).
- Poll `cloud-init status --wait` (or watch `/run/cloud-init/result.json`)
for the terminal state after boot.
- Across repeated runs of many (dozens to ~75) sequential VM boots from
the same base image in one batch, roughly **1-2% of individual VM boots**
hit this exact `IsADirectoryError`, always on the single **heaviest**
configuration in the batch (most RAM + most attached block devices) —
never on lighter configurations in the same batch, and never
consistently on the same position/index in the sequence from one batch
to the next.
- Confirmed under **two different VMM backends** (a KVM/QEMU-based one and
a Rust-based VMM) with the same guest image and same signature both
times — this rules out a VMM-specific cause; the failure is entirely
guest-side.
- All boots preceding the failing one in the same batch (i.e. earlier VMs
booted from the identical base image) complete cloud-init successfully,
which rules out the base/shared image itself already containing a
plain directory at this path — if it did, every VM booted from it would
fail identically and immediately, not roughly 1-in-50.

### A structural detail that may narrow the search

`cloudinit/cmd/main.py`'s `main_init()` is the same function for both the
`init-local` and `init` stages (differing only in `args.local`), and
**both invocations call `instancify()` → `_reflect_cur_instance()`**. In
every occurrence I've captured, the `init-local` stage's own
`_reflect_cur_instance()` call completes with **no recorded error** (see
`status.json` below — `init-local.errors` is empty), meaning a *valid*
symlink demonstrably exists at `/var/lib/cloud/instance` moments before
the crash. The crash is always in the **second**, network-stage `init`
call. So whatever replaces the symlink with a real directory does so in
the narrow window between `cloud-init-local.service` finishing and
`cloud-init.service`'s own second `instancify()` call — i.e., roughly the
window in which networking comes up and anything else ordered around that
point in boot runs. I have not been able to identify a specific writer
responsible for this in my own environment's guest-side code (see below).

## What I've ruled out in my own environment

- Not the VMM (reproduced under two different ones with identical guest
images).
- Not the shared/base image being corrupted (all prior boots in the same
batch from the identical image succeed).
- Not any first-party code in my own environment's guest-side stack — I
audited every guest-side process I control end-to-end (a boot-time
status-reporting agent that runs before `cloud-init-local.service`, and
a couple of provisioning helpers that run in the same early window) and
found none of them write to any path under `/var/lib/cloud/instance` —
only read from it (for diagnostics collection).
- I have not (yet) confirmed a `systemd-tmpfiles.d` rule or any other
OS/package-level cause; this remains a real candidate I haven't ruled
out.

## Environment details

- Cloud-init version: 22.3.4
- Operating System Distribution: Ubuntu 22.04 (jammy)
- Cloud provider, platform or installer type: private/on-prem virtualization
platform, `NoCloud` datasource fed via an attached seed device; not a
public cloud. Guest image is a stock Ubuntu 22.04 server cloud image with
guest agents added via an offline (`virt-customize`/chroot-based)
image-preparation step — no `cloud-init clean` or symlink manipulation
of `/var/lib/cloud` happens in that step for at least some of the
occurrences below (ruling out that specific offline-prep step as the
cause for those instances).

## cloud-init logs

Traceback (from the guest's `cloud-init.log`, one representative
occurrence; the exception and call path are byte-for-byte identical across
every occurrence I've captured):

```text
2026-06-22 15:32:26,559 - util.py[DEBUG]: Attempting to remove /var/lib/cloud/instance
2026-06-22 15:32:26,559 - util.py[WARNING]: failed stage init

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 767, in status_wrapper
ret = functor(name, args)
File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 419, in main_init
iid = init.instancify()
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 434, in instancify
return self._reflect_cur_instance()
File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 357, in _reflect_cur_instance
util.del_file(self.paths.instance_link)
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1910, in del_file
raise e
File "/usr/lib/python3/dist-packages/cloudinit/util.py", line 1907, in del_file
os.unlink(path)
IsADirectoryError: [Errno 21] Is a directory: '/var/lib/cloud/instance'
```

`/run/cloud-init/status.json` from the same occurrence:

```json
{
"v1": {
"init": {
"errors": ["[Errno 21] Is a directory: '/var/lib/cloud/instance'"],
"finished": 1782142346.5624669,
"start": 1782142346.5020645
},
"init-local": { "errors": [], "finished": 1782142344.5405004, "start": 1782142342.7768252 },
"modules-config": { "errors": [], "finished": null, "start": null },
"modules-final": { "errors": [], "finished": null, "start": null },
"modules-init": { "errors": [], "finished": null, "start": null }
}
}
```

I've captured this signature independently on four separate occasions over
roughly a month, always with this exact traceback and always on the
heaviest-configuration VM in a batch. In one of the four, `modules-init`/
`modules-config`/`modules-final` had actually run to completion before the
terminal `failed` state was recorded (the `init` stage's error is recorded
regardless of what runs afterward) — so the reliable fingerprint is
specifically the `init`-stage `IsADirectoryError`, not the
modules-never-ran pattern, which is incidental to whether anything
downstream had a chance to start before the terminal state was read.

## Proposed fix

I've prototyped hardening the four places in `cloudinit/stages.py`'s
`Init` class that assume `paths.instance_link` is always a symlink-or-
absent (`_reflect_cur_instance()`, `purge_cache(rm_instance_lnk=True)`,
and both `instance_link` removals in `_get_data_source()`) so that if the
path is ever found to be a real directory, it's healed (logged at
`WARNING` and removed recursively) instead of crashing — rather than
continuing to chase individual writers module-by-module, which is how
#3710's fix scope ended up not covering whatever caused this recurrence.
Opened #6980 with that change, now ready for review, independent of
whether the root cause of the promotion itself ever gets identified.

Start with Init in cloudinit/stages.py, especially _reflect_cur_instance(), purge_cache(), and _get_data_source(), then trace the init entry point in cloudinit/cmd/main.py. Review the prototype in #6980 and validate that a real /var/lib/cloud/instance directory is handled without aborting the init stage, with the failure no longer recorded.

Written by the indexing model from the issue text.