S3 export: uploads recorded as present without any content-integrity check (no ETag/MD5 comparison, HEAD-only presence check on fsck --fast)

# CORRECTION (2026-08-01)

> [!IMPORTANT]
> The original report below contained a **factual error** about how `fsck --from ` behaves on exported files, corrected by @joeyh in [this comment](https://github.com/con/git-annex/issues/284#issuecomment-5168164590). fsck **does** download and hash the object; it is not HEAD-only. See the [follow-up retraction comment](https://github.com/con/git-annex/issues/284#issuecomment-5169155633) and the strike-through edits below.
>
> The upload-side observations (no `Content-MD5` set, `InfoPresent` logged unconditionally, ETag never compared) are correct. The fsck-side framing was wrong.

---

## Summary

**Extra disclaimer as not posted from @yarikoptic-gitmate: it is not yet curated analysis by claude code, hinted on HI-to-HI communication, and my prior experiences in designing uploads for dandiarchive.org**

Interest triggered by a discussion with OpenNeuro folk (cc @effigies) about relying on `git annex fsck` to guarantee that files in an S3-backed archive are actually intact. ~~In the current S3 export code path, `fsck --from ` cannot deliver that guarantee — see below.~~ *[CORRECTION: fsck does verify content by download+hash even for exports — see [`Command/Fsck.hs:163-217`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Fsck.hs#L163-L217). The remaining concern is narrower: cost of periodic re-fsck on archive-scale exports, and the window between upload and first fsck during which `InfoPresent` is trusted.]*

When `git annex export` uploads a file to an S3 remote (`exporttree=yes`), git-annex marks the file as present on the remote as soon as the HTTP upload returns without an exception. There is **no end-to-end integrity check** between the file's content and what actually landed in the bucket:

- The `ETag` returned by `putObject` / `postCompleteMultipartUpload` is captured but never compared against anything ([`Remote/S3.hs:394`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L394), [`Remote/S3.hs:433`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L433); discarded at [`Remote/S3.hs:550`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L550) via `void $ storeExportS3'`).
- `cleanupExport` writes `InfoPresent` to the location log and updates the export DB unconditionally on non-exception return ([`Command/Export.hs:391-396`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Export.hs#L391-L396)).
- `checkPresentExportS3` on the authenticated code path only does a `HEAD` request and returns true if any object metadata is present — no size, no checksum, no ETag compare ([`Remote/S3.hs:542-547`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L542-L547), [`Remote/S3.hs:585-587`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L585-L587)). The public-URL fallback path *does* verify size via `Url.checkBoth u (fromKey keySize k)` at [`Remote/S3.hs:589-590`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L589-L590), so the authenticated (common) case is actually **weaker** than the anonymous one. Because the exported object is keyed by tree-path (not by git-annex key/hash), a bare `HEAD` cannot detect content mismatch even in principle. *[EDIT: this is about `checkPresent` in isolation — used by `whereis`-style queries and as fsck's first step. fsck itself does more than HEAD, see the CORRECTION above.]*
- Only `retrieveExportS3` verifies content ([`Remote/S3.hs:563-573`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L563-L573), via `verifyKeyContentIncrementally AlwaysVerify`) — but by then the false `InfoPresent` has already been recorded, and other copies may have been dropped based on it. *[EDIT: `retrieveExportS3` is also what `fsck --from` invokes, so fsck triggers this verification too. Under `--fast` mode however, [`Command/Fsck.hs:207-210`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Fsck.hs#L207-L210) skips the download and reverts to HEAD-only.]*

So a partial / truncated / silently-corrupted S3 upload (buggy S3-compatible endpoint, transient network corruption, a botched multipart complete, etc.) can be recorded as a valid copy and ~~only surface as an error much later, when the user tries to fetch it back~~ *[EDIT: … surface as an error only on the next full-download step — either a `git annex get`, or a `fsck --from` without `--fast`. During the window between upload and that next full-verify, `InfoPresent` is trusted for numcopies/whereis accounting.]*

## Realistic ~~failure~~ **dataloss** path

1. `git annex export --to s3remote`
2. `putObject` (or `postCompleteMultipartUpload`) returns HTTP 200 with an ETag; the object in S3 is truncated / corrupted / a leftover from a prior failed part upload.
3. `storeExportS3` returns normally → `cleanupExport` records `InfoPresent` for that key at the S3 remote's UUID.
4. ~~`git annex fsck --from s3remote` → `checkPresentExportS3` → `HEAD` → object exists → "OK".~~ *[CORRECTION: `git annex fsck --from s3remote` (without `--fast`) downloads the object via `retrieveKeyFile` and hashes it against the key. Corruption is detected here — this step catches the fault. Under `--fast`, however, only the HEAD check runs and corruption goes undetected. So the failure path applies to (a) users who don't run `fsck` between the upload and their next drop-based-on-numcopies decision, and (b) users who run `fsck --from --fast`.]*
5. User trusts S3 as a valid copy; local content is eventually dropped elsewhere on the strength of that.
6. Later `git annex get --from s3remote` finally trips `verifyKeyContentIncrementally` and discovers the hash mismatch — after the safety margin has already been eroded. *[EDIT: or, equivalently, `git annex fsck --from s3remote` (non-`--fast`) discovers it, at the cost of a full download of the object.]*

Related but distinct from the well-known "S3 doesn't guarantee the ETag equals MD5 for encrypted / multipart objects" caveat — the issue here is that git-annex isn't attempting *any* comparison at all.

## Suggested mitigations

Both strategies below are used in production by [dandi/dandi-cli](https://github.com/dandi/dandi-cli). Either or both could apply to git-annex's S3 export path:

1. **Ask S3 to validate on its side via `Content-MD5` / SigV4 payload hash.** For single-part upload, git-annex knows (or can compute during upload) the MD5 of the object and can pass it as `Content-MD5`; S3 will reject the PUT with `BadDigest` if the body doesn't match. dandi-cli does exactly this for the small-file path — see `dandi/files/zarr.py` around L940 in [`_upload_zarr_file`](https://github.com/dandi/dandi-cli/blob/master/dandi/files/zarr.py#L934-L975):
```python
headers = {"Content-MD5": item.base64_digest}
...
storage_session.put(upload_url, data=fp, ..., headers=headers, ...)
```
The `aws` Haskell library exposes `poContentMD5` on `PutObject` for exactly this. Same idea for `S3.uploadPart`.

2. **Reconstruct and compare the multipart ETag client-side.** git-annex fully controls the multipart flow ([`Remote/S3.hs:396-435`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L396-L435)) — it can accumulate the per-part MD5s during upload, compute `md5(concat(part_md5s))` and append `-`, and compare that against `S3.cmurETag` returned by `postCompleteMultipartUpload`. Mismatch → fail the export before `cleanupExport` runs.

This is exactly what dandi-cli's `DandiETag` does — the algorithm lives in [`dandischema/digests/dandietag.py`](https://github.com/dandi/dandischema/blob/master/dandischema/digests/dandietag.py) (`DandiETag` class at L96, `as_str()` at L132-140: `f"{md5(concat(md5(part_i))).hexdigest()}-{len(parts)}"`), and the comparison against the server-returned ETag is in [`dandi/files/bases.py`](https://github.com/dandi/dandi-cli/blob/master/dandi/files/bases.py) L373-471, boiling down to:
```python
etagger = get_dandietag(self.filepath) # bases.py:374
filetag = etagger.as_str() # bases.py:375
...
final_etag = rxml.findtext(f"{ns}ETag").strip('"') # bases.py:463-465
if final_etag != filetag:
# bail out — server and client disagree on final ETag (bases.py:466-471)
```
Per-part sizes are also cross-checked against `etagger.get_part(...)` before upload (`bases.py` around L681).

**Note that mitigation (a) subsumes (b)** at the S3-transit level: if each part's `upContentMD5` is validated server-side, the combined ETag mismatch cannot occur (barring post-ack corruption on the S3 side, which is a different threat model). Recommend (a) alone for the S3 code path — it's a smaller, more robust patch.

**Library availability check:** [`git-annex.cabal` line 296](https://github.com/con/git-annex/blob/ccf99dd4fe/git-annex.cabal#L296) requires `aws >= 0.24.1`; [`stack.yaml`](https://github.com/con/git-annex/blob/ccf99dd4fe/stack.yaml) currently resolves `aws-0.25.2`. Both `poContentMD5 :: Maybe (Digest MD5)` on `PutObject` and `upContentMD5 :: Maybe (Digest MD5)` on `UploadPart` are present at that version, so mitigation (a) needs **no library bump** — just wiring an MD5 computed alongside the existing stream through `httpBodyStorer` / `handlePopper`. The current `putObject` helper at [`Remote/S3.hs:1107-1116`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L1107-L1116) explicitly leaves `poContentMD5 = Nothing` (populates only storage class / metadata / auto-make-bucket / ACL / tagging).

3. **The correct semantic fix, longer-term:** neither (a) nor (b) validates against the *git-annex key hash*. They only prove "what S3 stored matches what we sent" — not "what we sent matches the key". A full end-to-end fix would hash the file with the key's own backend (SHA256E, etc.) during the upload stream and refuse to record `InfoPresent` on a mismatch. This is more invasive but is what the key-hash contract implies.

Even without any of the above, at minimum: if verification isn't structurally possible for a given endpoint/backend, refuse to record `InfoPresent` after export without an explicit verify-roundtrip (or gate on an option), so users at least have a way to opt into "upload-then-verify" semantics.

## Impact

Silent-corruption exposure for any git-annex user relying on an S3 remote as a durable copy of an exported tree. Especially concerning for S3-compatible (non-AWS) endpoints where transport-layer integrity guarantees are weaker than on AWS proper. ~~And — bringing it back to the trigger — `fsck --from ` **cannot** currently be used to reassure downstream users (e.g. OpenNeuro) that archived files are intact; it only confirms that *some* object with the right name exists.~~ *[CORRECTION: fsck DOES download and hash for exports — it can be used to reassure downstream users that archive files are intact, at the cost of downloading the archive (or the delta since the last incremental-fsck pass). The remaining concern is the download cost for archive-scale exports and the window between upload and the next full-verify.]*

Code references (git-annex 10.20260717, master @ ccf99dd4fe)

**Upload** — `Remote/S3.hs`:

- [L386-395](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L386-L395) single-part: `S3.porETag resp` returned but never validated.
- [L396-435](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L396-L435) multipart: per-part `S3.uprETag` and final `S3.cmurETag` returned but never validated.
- [L549-561](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L549-L561) `storeExportS3`: `void $ storeExportS3'` — ETag and version-id both discarded.

**Location log** — `Command/Export.hs`:

```haskell
-- Command/Export.hs:391-396
cleanupExport r db ek loc sent = do
liftIO $ addExportedLocation db ek loc
when (sent && not (isGitShaKey ek)) $
logChange NoLiveUpdate ek (uuid r) InfoPresent
return True
```

Called from `performExport` ([`Command/Export.hs:312-330`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Export.hs#L312-L330)) on any non-exception return from `storer`.

**Presence check** — `Remote/S3.hs`:

```haskell
-- Remote/S3.hs:542-547
checkKeyHelper' info h o limit = liftIO $ runResourceT $ do
rsp <- sendS3Handle h req
extractFromResourceT (isJust $ S3.horMetadata rsp)
where
req = limit $ S3.headObject (bucket info) o

-- Remote/S3.hs:585-591
checkPresentExportS3 hv r info k loc = withS3Handle hv $ \case
Right h -> checkKeyHelper info h (Left (T.pack $ bucketExportLocation info loc))
...
```

**Retrieval** (verifies content; called by `git annex get` AND by `fsck --from` in the non-`--fast` case) — [`Remote/S3.hs:563-573`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L563-L573):

```haskell
retrieveExportS3 hv r info k loc f p =
verifyKeyContentIncrementally AlwaysVerify k $ \iv ->
withS3Handle hv $ \case
Right h -> retrieveHelper (gitconfig r) info h (Left (T.pack exportloc)) f p iv
...
```

**Fsck-from-remote flow** ([`Command/Fsck.hs:163-217`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Fsck.hs#L163-L217)) — `performRemote` first calls `Remote.hasKey` (HEAD-only for exports), then downloads via `Remote.retrieveKeyFile … (RemoteVerify remote)` at [line 214](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Fsck.hs#L214). For exports, `retrieveKeyFile` is adjusted at [`Remote/Helper/ExportImport.hs:239-244`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/Helper/ExportImport.hs#L239-L244) to route through `retrieveFromImportOrExport` → `retrieveExport` above. `--fast` mode skips the download at [`Command/Fsck.hs:207-210`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Fsck.hs#L207-L210).

Related sites in the same code path

Adjacent places that inherit or extend the same "trust the HTTP response" pattern — worth touching in the same fix:

- **`storeExportWithContentIdentifierS3`** ([`Remote/S3.hs:796-807`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L796-L807)) *does* retain the ETag, but only as an opaque content-identifier — never compared against an expected content hash. A corrupt upload therefore gets a stable-but-meaningless CID; `checkPresentExportWithContentIdentifierS3` ([`Remote/S3.hs:819-825`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L819-L825)) will then succeed forever on a HEAD that matches that same corrupt ETag via `limitHeadToContentIdentifier`. Also note the dead-branch at [lines 797-799](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/S3.hs#L797-L799): both `versioning info` and `otherwise` branches do the same `go` — probably an unfinished intention to differentiate versioned vs unversioned verification (versioned case ought to be able to do a version-id round-trip verify).
- **Annex-object → export rename shortcut** ([`Command/Export.hs:372-388`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Export.hs#L372-L388), `tryrenameannexobject`): when the annex-object is already known to be present on the remote, git-annex issues a server-side S3 copy (`renameExport`) instead of re-uploading. It then calls `addExportedLocation` unconditionally on success ([line 382](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Export.hs#L382)). This inherits any pre-existing corruption at the source key without noticing.
- ~~No fsck / verify hook currently touches the export path (`grep -n 'export\|Export' Command/Fsck.hs` → nothing; `Logs/Export.hs` and `Logs/Export/Pure.hs` manage the export tree log without triggering content verification). So there is currently no post-hoc way for `fsck --from ` to catch a corrupt export short of downloading and re-hashing.~~ *[CORRECTION: downloading and re-hashing is exactly what fsck does. The routing goes via `retrieveKeyFile` (adjusted for exports by [`Remote/Helper/ExportImport.hs:239-244`](https://github.com/con/git-annex/blob/ccf99dd4fe/Remote/Helper/ExportImport.hs#L239-L244)), not via a fsck-specific export hook.]*

Notes / caveats

- With SSE-KMS / SSE-C / CSE, the returned ETag is not the MD5 of the plaintext object, so the *client-side ETag reconstruction* (mitigation b) needs to account for that. The server-side `Content-MD5` header (mitigation a) still works fine — S3 verifies the pre-encryption body.
- `InfoPresent` is skipped for `isGitShaKey ek` ([`Command/Export.hs:394`](https://github.com/con/git-annex/blob/ccf99dd4fe/Command/Export.hs#L394)), i.e. for non-annexed git blobs being exported. For annexed keys — the case that matters for durability guarantees — `InfoPresent` is written unconditionally on any non-exception return of the storer.
- The same reasoning likely applies to other export-capable remotes (`Remote/WebDAV.hs`, `Remote/Rclone.hs`, …) — but this issue is scoped to the S3 code path we actually read.

---

*Reported after a static code review of the S3 export path (`Remote/S3.hs`, `Command/Export.hs`) at master `ccf99dd4fe`, followed by an independent second-pass verification. Not a live reproducer. Filed with the `ai-uncurated` label per repo convention; please double-check line numbers / logic before treating as authoritative. **Edited 2026-08-01** to correct incorrect claims about `fsck --from` behaviour after @joeyh's [correction](https://github.com/con/git-annex/issues/284#issuecomment-5168164590); see the [retraction comment](https://github.com/con/git-annex/issues/284#issuecomment-RETRACTION) for details.*

Start with Remote/S3.hs at storeExportS3, the single- and multipart upload paths, checkPresentExportS3, and the putObject helper. Then read Command/Export.hs at cleanupExport to understand when InfoPresent is recorded. Done means the agreed upload-integrity behavior is implemented without recording a valid export prematurely, with the existing fsck and export behavior checked against the issue’s correction.

Written by the indexing model from the issue text.