con / con/git-annex

httpalso layout discovery short-circuits on Right False, breaks fsck for non-first layouts

Open
#275 0 comments 0 reactions 0 assignees View on GitHub
ai-uncurated bug regression
Dominant language
Python
Stars
29
Forks
5
PR merge metrics
No merged PRs in 30d

Description

After commit 5d1baa5026 *"httpalso: Fix bugs in handling content not being
present on the remote"* (released in `10.20260316`), httpalso layout
discovery no longer falls back when the first probed layout returns
`Right False` (HTTP 404). `fsck --from ` falsely reports content
missing when the content lives under a non-first layout.

```
> git -C main annex fsck --from=httpalso_bad # content stored as flat baseurl/
fsck file.dat (httpalso_bad)
** Failed to verify content of file.dat (httpalso_bad):
expected to be present, but its content is missing
```

A control httpalso remote with content stored at the first-tried
`hashDirLower(2)` layout works fine. The HTTP access log shows exactly one
HEAD request to the first layout (404) — no fallback HEAD to other layouts.

Root cause

5d1baa5026 widened the downloader return type in `keyUrlAction` from
`Either String ()` to `Either String a`. The existing iterator in
`Remote/HttpAlso.hs:177-185`:

```haskell
go' (k, urlcontent) = \case
[] -> ...
(mklayout:rest) -> do
r <- ...
case r of
Right _ -> ... -- ← short-circuits on Right False too
Left _ -> go' (k, urlcontent) rest
```

was correct when `a = ()` (only success was representable), but
`checkKey'` at `Remote/HttpAlso.hs:143-145` returns `Right Bool`, so
`Right False` (definitive "content absent at this layout") now also
short-circuits and prevents fallback.

Suggested fix: distinguish layout-probe failure (try next layout) from
content-absence (definitive answer), e.g.:

```haskell
case r of
Right True -> ... -- found
Right False -> go' (k, urlcontent) rest -- try next layout
Left _ -> go' (k, urlcontent) rest
```

Reproducer (POSIX shell, exits non-zero when bug fires)

```
#!/bin/sh
# SPDX-FileCopyrightText: 2026 Yaroslav Halchenko
# SPDX-License-Identifier: MIT
#
# Reproducer for the git-annex httpalso layout-discovery bug introduced in
# commit 5d1baa5026 ("httpalso: Fix bugs in handling content not being
# present on the remote", first released in 10.20260316).
#
# Claim: keyUrlAction in Remote/HttpAlso.hs treats a confident "Right False"
# (HTTP 404 -- content not present at the tried URL) the same way as a
# successful "Right True" (content present): it stops iterating the
# supported layouts at the first definitive answer. Therefore if content
# actually lives under a non-first layout, checkpresent locks onto the
# first layout's 404 and reports the content as absent even though it is
# reachable at a later layout's URL.
#
# Strategy:
# * primary remote: a "directory" special remote (purely so that
# `git annex initremote --sameas` is satisfied). It does NOT contain
# the actual data.
# * publish a single annex key on a local HTTP server under the FLAT
# baseurl/ layout (the SECOND outer entry in
# supportedLayouts). The FIRST outer entry (hashDirLower /
# hashDirMixed at HashLevels 2) therefore returns HTTP 404 ->
# Right False.
# * verify with a control: also stage the same data under the first
# layout in a SEPARATE HTTP root + httpalso remote, and confirm
# fsck succeeds there.
# * verify with the experiment: against the flat-only HTTP root,
# a fixed httpalso would iterate to the flat layout and locate the
# content; the buggy one stops at the first Right False.
#
# Exits non-zero when the bug is observed.

set -eux
PS4='> '

TMP="$(mktemp -d)"
# pick a port unlikely to collide with anything else, parameterised on PID
PORT_BAD=$(( 30000 + ($$ % 15000) ))
PORT_OK=$(( PORT_BAD + 1 ))

# 1) Source repo: stage one annexed file, capture its key.
git init -q -b main "$TMP/src"
cd "$TMP/src"
git config user.email t@t
git config user.name t
git annex init -q src
printf 'hello-httpalso\n' > hello.txt
git annex add -q hello.txt
git commit -q -m add
KEY="$(git annex lookupkey hello.txt)"
test -n "$KEY"
CONTENT="$(git annex contentlocation "$KEY")"

# 2a) BAD layout root: publish the key ONLY under the flat baseurl/
# layout. Layout #1 (hashDirLower(2)) will 404.
SRV_BAD="$TMP/srv-bad"
mkdir -p "$SRV_BAD"
cp "$TMP/src/$CONTENT" "$SRV_BAD/$KEY"

# 2b) OK layout root: publish the key under hashDirLower(2)//
# (the very first layout supportedLayouts tries).
SRV_OK="$TMP/srv-ok"
# Compute the hashDirLower(2) path the way git-annex does. The Haskell
# code: baseurl toInternalGitPath (hashDirLower (HashLevels 2) k)
# keyFile k keyFile k.
#
# Instead of replicating the hash math here, fish it out of git-annex
# itself via examinekey.
HASHDIR2_LOWER="$(git annex examinekey --format='${hashdirlower}' "$KEY")"
KEYDIR="$SRV_OK/${HASHDIR2_LOWER}${KEY}"
mkdir -p "$KEYDIR"
cp "$TMP/src/$CONTENT" "$KEYDIR/$KEY"
ls -R "$SRV_OK" | head -20

# 3) Start two HTTP servers; wait for each.
cd "$SRV_BAD"
python3 -m http.server "$PORT_BAD" --bind 127.0.0.1 >"$TMP/http_bad.log" 2>&1 &
HTTP_BAD_PID=$!
cd "$SRV_OK"
python3 -m http.server "$PORT_OK" --bind 127.0.0.1 >"$TMP/http_ok.log" 2>&1 &
HTTP_OK_PID=$!
i=0
until curl -fsS "http://127.0.0.1:$PORT_BAD/$KEY" -o /dev/null \
&& curl -fsS "http://127.0.0.1:$PORT_OK/${HASHDIR2_LOWER}${KEY}/$KEY" -o /dev/null; do
i=$((i+1))
if [ $i -ge 80 ]; then
echo "one or both servers did not start"
cat "$TMP/http_bad.log" "$TMP/http_ok.log"
kill $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null
exit 2
fi
sleep 0.1
done

# 4) Consumer repo with a directory primary and TWO httpalso aliases:
# httpalso_bad -> flat-layout server, httpalso_ok -> hashDir-layout server.
git init -q -b main "$TMP/dst"
cd "$TMP/dst"
git config user.email t@t
git config user.name t
git annex init -q dst
git config annex.security.allowed-http-addresses all
git config annex.security.allowed-ip-addresses all

DIRREMOTE="$TMP/dirstore"
mkdir -p "$DIRREMOTE"
git annex initremote -q dirstore type=directory directory="$DIRREMOTE" encryption=none

git annex initremote -q httpalso_ok \
--sameas=dirstore \
type=httpalso url="http://127.0.0.1:$PORT_OK/" \
encryption=none

git annex initremote -q httpalso_bad \
--sameas=dirstore \
type=httpalso url="http://127.0.0.1:$PORT_BAD/" \
encryption=none

# Bring in the file pointer and the git-annex branch.
git remote add src "$TMP/src"
git fetch -q src
git checkout -q src/main -- hello.txt
git annex sync -q --no-push --no-pull || true

# Declare the key as present on the shared (dirstore/httpalso_*) UUID.
DIR_UUID="$(git config remote.dirstore.annex-uuid)"
test -n "$DIR_UUID"
git annex setpresentkey "$KEY" "$DIR_UUID" 1

# 5) CONTROL: fsck via httpalso_ok must succeed (first layout matches).
set +e
git annex fsck --from httpalso_ok hello.txt > "$TMP/fsck_ok.log" 2>&1
FSCK_OK_RC=$?
set -e
echo "=== fsck_ok.log (control) ==="; cat "$TMP/fsck_ok.log"; echo "============================"

# 6) EXPERIMENT: fsck via httpalso_bad. With the bug, this fails because
# the first layout 404s, returns Right False, and the iterator stops.
set +e
git annex fsck --from httpalso_bad hello.txt > "$TMP/fsck_bad.log" 2>&1
FSCK_BAD_RC=$?
set -e
echo "=== fsck_bad.log (experiment) ==="; cat "$TMP/fsck_bad.log"; echo "================================="

# Show which URLs httpalso actually requested on each server.
echo "=== http_bad.log ==="; cat "$TMP/http_bad.log"; echo "===================="
echo "=== http_ok.log ==="; cat "$TMP/http_ok.log"; echo "===================="

# Shut down HTTP servers (no trap per convention).
kill $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null || true
wait $HTTP_BAD_PID $HTTP_OK_PID 2>/dev/null || true

# 7) Verdict.
# Control should not show a failure; if it does, our setup is wrong
# and we cannot make a claim.
if [ $FSCK_OK_RC -ne 0 ]; then
echo "INDETERMINATE: control fsck (against the hashDir-layout server) failed; setup is wrong."
exit 2
fi
if grep -qE 'fsck: [1-9]|expected to be present, but its content is missing' "$TMP/fsck_ok.log"; then
echo "INDETERMINATE: control fsck did not see content; setup is wrong."
exit 2
fi

# Experiment failing the same way the bug predicts.
if [ $FSCK_BAD_RC -ne 0 ] || grep -qE 'fsck: [1-9]|expected to be present, but its content is missing' "$TMP/fsck_bad.log"; then
echo "BUG REPRODUCED: httpalso layout iterator stopped at first Right False (HTTP 404) and did not fall back to the flat layout where content lives."
exit 1
fi
echo "NOT REPRODUCED: httpalso located content despite first-layout 404."
exit 0

```

See `reproducer-httpalso-layout-lockin.sh` — runs `python3 -m http.server`,
stages two httpalso remotes (one with content at `hashDirLower(2)/...` as a
control, one with content at flat `baseurl/`), runs `fsck --from=` on
each. Control passes (exit 2 if it fails), experiment fails (exit 1) → bug
reproduced.

Regression evidence

Before commit 5d1baa5026 the iterator's downloader returned `Either String ()`.
Only `Right ()` was representable; the `Right _` short-circuit was correct.
The reproducer's `fsck --from=httpalso_bad` (content stored at flat
`baseurl/`) would have succeeded by falling back through layouts on
`Left _`.

`5d1baa5026` widened the return type to `Either String a`. In `checkKey'`
this `a` is `Bool` — and `Right False` (definitive "absent at this layout")
now also short-circuits in the iterator, preventing fallback.

The commit's stated purpose was *"httpalso: Fix bugs in handling content not
being present on the remote"* — i.e. it intentionally wanted to distinguish
absent-from-this-layout from error. But the iterator at lines 177-185 was not
updated to match the wider return type. The release notes for `10.20260316`
describe only the intended fix:

> httpalso: Fix bugs in handling content not being present on the remote.

The breaking of layout fallback is not mentioned and is clearly an
unintended side effect.

Version

```
Verified with: git-annex 10.20260421
Confirmed present in: 10.20260525 (Remote/HttpAlso.hs untouched in the interim)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.