con / con/git-annex

`git annex test` still fails on BeeGFS: `renamePath ... resource busy` in the export tests

Open
#288 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
29
Forks
5
PR merge metrics
No merged PRs in 30d

Description

We now have BeeGFS under continuous CI, and the export-related `rename`
failure that was reported and closed a while back is reproducing again on
current daily builds — on **both** BeeGFS 7.4.x and 8.x.

Filing here rather than upstream because this is where the daily builds
under test are produced, and because the reproducer is a GitHub Actions
job that could be adopted here directly.

## What fails

git-annex **10.20260717** (daily build from this repo), BeeGFS client
**7.4.6**, `HOME` and `TMPDIR` on the BeeGFS mount:

```
export and import: FAIL (5.41s)
./Test/Framework.hs:100:
export to dir failed with unexpected exit code (transcript follows)
mv: cannot move '.git/annex/othertmp/07cdc877-0d48-4a.0/07cdc877-0d48-4a' to '.git/annex/export.ex/07cdc877-0d48-4a88-b266338-0': Device or resource busy
mv: cannot move '.git/annex/othertmp/07cdc877-0d48-4a.0/07cdc877-0d48-4a' to '.git/annex/export.ex/07cdc877-0d48-4a88-b266338-1': Device or resource busy
git-annex: renamePath:rename '.git/annex/othertmp/07cdc877-0d48-4a.0/07cdc877-0d48-4a' to '.git/annex/export.ex/07cdc877-0d48-4a88-b107-6c53ed06d1ab': resource busy (Device or resource busy)
HasCallStack backtrace:
collectBacktraces, called at libraries/ghc-internal/src/GHC/Internal/Exception.hs:169:13 in ghc-internal:GHC.Internal.Exception
toExceptionWithBacktrace, called at libraries/ghc-internal/src/GHC/Internal/IO.hs:260:11 in ghc-internal:GHC.Internal.IO
throwIO, called at libraries/exceptions/src/Control/Monad/Catch.hs:308:12 in exceptions-0.10.9-inplace:Control.Monad.Catch
throwM, called at Messages.hs:387:17 in git-annex-10.20260717-...-git-annex
```

and, in a second suite pass:

```
export and import of subdir: FAIL (5.38s)
export of subdir failed with unexpected exit code (transcript follows)
mv: cannot move '.git/annex/othertmp/bf846fa6-ad01-48.0/bf846fa6-ad01-48' to '.git/annex/export.ex/bf846fa6-ad01-485e-9280240-1': Device or resource busy
...
git-annex: renamePath:rename '...' to '...': resource busy (Device or resource busy)
```

Both are in `Repo Tests v10 locked`; each pass reported `1 out of 13
tests failed`. A third pass in the same job was all green, so it is not
every rename — it is specifically the `othertmp` -> `export.ex` move.

## Why this is probably not "just a broken filesystem"

The same harness runs three other suites on the *same* mount, and this
is what separates them:

| Suite on BeeGFS | 7.4.6 | 8.1.0 |
| --- | --- | --- |
| `git annex test` | **FAIL** | **FAIL** |
| git's own testsuite (`t0*.sh t1*.sh`, ~10k assertions) | pass | pass |
| stress-ng (20 filesystem stressors, `--verify`, incl. `rename`) | pass | pass |
| pjdfstest (POSIX conformance, ~8800 assertions) | FAIL | FAIL |

Git's entire index/refs/object-store test range passes on BeeGFS, and
stress-ng's `rename` stressor passes with verification on. So whatever
BeeGFS is doing here is not something that breaks git generally — it is
reachable specifically through the pattern git-annex's exporter uses.

pjdfstest does fail on BeeGFS, so there are genuine POSIX conformance
gaps; correlating its failing assertions against this rename pattern is
the obvious next step and I have not done it yet.

## Reproducing

The whole thing is one command via the harness in
[con/eval-under](https://github.com/con/eval-under):

```bash
# stands up a throwaway BeeGFS cluster (docker compose: mgmtd+meta+storage),
# mounts it, runs the suite with TMPDIR/HOME on the mount, tears down
sudo bin/ci/run-under.sh beegfs 7.4.6 git-annex
```

or directly:

```bash
sudo bin/eval-under beegfs --set-home -- git annex test
```

Environment: `ubuntu-22.04` runner, BeeGFS client via DKMS, all services
on localhost over TCP. Mount comes up clean and unmounts clean — dmesg in
the job log shows `BeeGFS mount ready` and a clean `BeeGFS unmounted`, so
this is not a half-configured mount.

## Runs and full output

From run [`32524403363`](https://github.com/con/eval-under/actions/runs/32524403363)
(master, 2026-08-21):

- BeeGFS 7.4.6 / `git annex test` — **FAIL** — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903304981) (transcript above)
- BeeGFS 8.1.0 / `git annex test` — **FAIL** — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903305062)
- BeeGFS 7.4.6 / git testsuite — pass — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903304994)
- BeeGFS 8.1.0 / git testsuite — pass — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903305198)
- BeeGFS 7.4.6 / pjdfstest — FAIL — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903304985)
- BeeGFS 8.1.0 / pjdfstest — FAIL — [job log](https://github.com/con/eval-under/actions/runs/32524403363/job/96903305024)

Live grid across all backends: https://con.github.io/eval-under/ —
each cell links to its own job log. Per-backend mount/export settings and
the standing explanation for every known-red cell are in
[GOTCHAS.md](https://github.com/con/eval-under/blob/master/GOTCHAS.md).

This runs weekly, so the state stays current rather than being a
one-off snapshot.

## Prior art

- Original report: [35 failed tests on beegfs](https://git-annex.branchable.com/bugs/35_failed_tests_on_beegfs/)
- Follow-up, since closed: [beegfs: init tests FAIL resource busy](https://git-annex.branchable.com/projects/dandi/bugs-done/beegfs__58___init_tests_FAIL_resource_busy/)
(filed under `projects/dandi/bugs-done/`, i.e. marked resolved — with
the caveat recorded there that BeeGFS was not actually installed to
verify the fix)

> **TODO before filing:** link the specific commit that closed the
> follow-up bug. I could not retrieve it — this session's egress proxy
> blocks `git-annex.branchable.com`, and upstream git-annex source was
> not reachable either, so rather than guess at a hash I have left this
> for you to fill in from the bug page.

Given the fix was made without a BeeGFS system to test against, the most
likely readings are that it addressed a different rename path than the
exporter's, or that it regressed. Either way there is now a standing
reproducer.

## What would help

1. Confirmation of which commit closed the earlier bug, so we can check
whether this path was ever covered.
2. Whether the `othertmp` -> `export.ex` move is expected to be a plain
`rename(2)`, or whether a link+unlink fallback would be acceptable on
filesystems that return `EBUSY` here.
3. If useful, we can add BeeGFS as a flavour to this repo's build matrix
— the harness is already a self-contained script pair and the CI cell
takes ~10 minutes.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the con/eval-under harness, especially bin/ci/run-under.sh and bin/eval-under, and reproduce the failure with BeeGFS using git annex test. Compare the exporter’s othertmp-to-export.ex move with the failing pjdfstest results and review GOTCHAS.md and the linked CI logs. Done means the cause and the earlier fix are identified, and the export tests pass on the affected BeeGFS versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker-compose, git, github-actions, linux
Domain
ci-cd, operating-systems, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.