files write: support add-compatible DAG settings (layout/chunker); repeated --truncate rewrites are non-deterministic and leave zero-length blocks
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
### Checklist
- [x] My issue is specific & actionable.
- [x] I am not suggesting a protocol enhancement.
- [x] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.
### Description
## Summary
`ipfs files write` always builds the file DAG with the **trickle** layout, so for anything larger than a single chunk the same bytes get a different CID through `files write` than through `ipfs add` (which defaults to the **balanced** layout). And once the destination file already exists, repeated identical writes with `--truncate` are **non-deterministic** (a different CID on every rewrite), leave **zero-length blocks** in the resulting DAG, and occasionally produce an **empty file**.
I would like a way to make `files write` produce CIDs identical to `ipfs add` — semantically equivalent to the sequence below, but as a single atomic operation:
```bash
cid=$(ipfs add --quieter --pin=false b.txt) # canonical CID for these bytes
ipfs files rm /a/b.txt # if present
ipfs files cp /ipfs/$cid /a/b.txt # MFS entry now carries the add-identical CID
```
The determinism, zero-block and atomicity problems below are worth fixing regardless of which layout is used.
## Background
The CID mismatch is documented in `ipfs files write --help`:
> ```
> WARNING:
>
> The CID produced by 'files write' will be different from 'ipfs add' because
> 'ipfs files write' creates a trickle-dag optimized for append-only operations.
> See '--trickle' in 'ipfs add --help' for more information.
>
> NOTE: The 'Import.UnixFSFileMaxLinks' config option does not apply to this command.
> Trickle DAG has a fixed internal structure optimized for append operations.
> To use configurable max-links, use 'ipfs add' with balanced DAG layout.
> ```
At the code level, the MFS write path goes through the DAG modifier in [`boxo/ipld/unixfs/mod/dagmodifier.go`](https://github.com/ipfs/boxo/blob/main/ipld/unixfs/mod/dagmodifier.go),which appends new data via `trickle.Append` — the layout is not configurable.
Since v0.40, the [`Import.*`](https://github.com/ipfs/kubo/blob/master/docs/config.md#import) config section (implementing [IPIP-499: UnixFS CID Profiles](https://specs.ipfs.tech/ipips/ipip-0499/)) exposes the DAG-shaping defaults for data ingestion. Current coverage:
| Option | Commands affected | Default |
|---|---|---|
| `Import.UnixFSDAGLayout` | `ipfs add` | `balanced` |
| `Import.UnixFSChunker` | `ipfs add` | `size-262144` |
| `Import.UnixFSRawLeaves` | `ipfs add`, `ipfs files write` | depends on CID version |
| `Import.CidVersion` / `Import.HashFunction` | `ipfs add` (MFS preserves them since v0.41) | `0` / `sha2-256` |
So part of the machinery already applies to `files write` (raw leaves, CID builder), but **layout and chunker cannot be controlled**, and the hardcoded MFS default is `trickle` — the opposite of `add`'s `balanced` default. The command's own help steers users towards `ipfs add` whenever CID stability matters (the WARNING above), which confirms there is no supported way to get an `add`-identical CID out of `files write` itself and `add --to-files`, the suggested escape hatch, cannot fill this role either (see Workaround).
## Reproduction
Kubo v0.43.0, linux-amd64, default configuration, default `--flush=true`. Run the following pair four times in a row — the piped-in content is identical every time (a payload of just a few bytes):
```bash
echo 123 | ipfs files write /a/b.txt --create=true --truncate=true --parents=true
ipfs files stat /a/b.txt
```
Observed:
1. **Only the first CID matches `ipfs add`.** After the very first run, the CID is exactly the one `ipfs add --quieter --pin=false` produces for the same bytes — the payload fits in a single chunk, so the trickle-vs-balanced difference is not visible yet. Every subsequent run of the same pair changes the CID again: the second, third and fourth runs each ended up with a different CID, none of them equal to the `add` CID or to each other. The CID of the file appears to depend on the previous DAG state at `/a/b.txt`, not only on the bytes written and the configured settings. (For payloads larger than one chunk, even the first `files write` diverges from `ipfs add` because of the hardcoded trickle layout — see Background.)
2. **Zero-length blocks.** The resulting DAG contains multiple zero-length child blocks (visible via `ipfs dag get` on the file CID and `ipfs block stat` on its children), even though the entire file is only a few bytes long. `--truncate` does not cleanly discard the previous DAG structure.
3. **Occasional empty file.** Intermittently, a run leaves `/a/b.txt` as an empty (0-byte) file. This suggests the write is not atomic — an interrupted or racy write can expose an empty target instead of either the old or the new content.
For comparison, the canonical CID for the same bytes is produced by:
```bash
echo 123 | ipfs add --quieter --pin=false
```
That value matches the CID of the **first** `files write` run above — and of none of the later ones. It is the CID that other peers and other `ipfs add` users expect for this content.
## Expected behavior
1. With equivalent settings (layout, chunker, CID version, raw leaves), `files write --create --truncate` produces the same CID as `ipfs add` for the same bytes.
2. Determinism: the CID depends only on the written bytes and the configured settings — never on what previously existed at the destination path.
3. `--truncate` fully discards the previous DAG; no zero-length blocks are left behind in the new one.
4. Atomicity: readers observe either the previous content or the new content — never an empty or partial file; a failed write leaves the previous file intact.
## Proposal
Extend the existing `Import` machinery rather than inventing a parallel one. Two wiring options (maintainers' choice):
**Option A — extend `Import.*` to `files write`:**
```bash
ipfs config --json Import.UnixFSDAGLayout '"balanced"'
ipfs config --json Import.UnixFSChunker '"size-262144"'
```
Both options would gain `ipfs files write` in their "Commands affected" list. Because flipping the MFS default from `trickle` to `balanced` would change CIDs for existing MFS users, MFS could keep `trickle` as its default and only follow `Import.UnixFSDAGLayout` when it is set explicitly.
**Option B — dedicated keys with current-behavior defaults:**
```bash
ipfs config --json Files.Write.Layout '"balanced"'
ipfs config --json Files.Write.Chunker '"size-262144"'
```
**Per-call flags** (mirroring `ipfs add`) in either case:
```bash
ipfs files write --layout=balanced --chunker=size-262144 --create --truncate --parents /a/b.txt
```
**Semantics / implementation note.** For `--truncate` (full-overwrite) writes under the balanced layout, the natural implementation is the workaround sequence executed inside the daemon: build the complete DAG with the same importer `ipfs add` uses (no extra pin — the MFS root already references it), then swap the new node into the MFS entry under the existing pin lock (the same lock v0.43 introduced to protect MFS writes from GC). This guarantees `add`-identical CIDs by construction and makes the overwrite atomic. Appends without `--truncate` can keep the trickle path (documented as such), since balanced DAGs are not append-friendly.
## Workaround
```bash
cid=$(echo 123 | ipfs add --quieter --pin=false)
ipfs files rm /a/b.txt 2>/dev/null || true
ipfs files cp /ipfs/$cid /a/b.txt --parents
```
`ipfs add --to-files` looks like a shorter alternative, but it does not remove any of the steps:
- **No stdin.** It requires a named file argument; piping is rejected with `to-files: cannot add unnamed files to MFS` ([`core/commands/add.go`](https://github.com/ipfs/kubo/blob/master/core/commands/add.go)).
- **No overwrite.** If the target name already exists in MFS, the add fails (`ErrDirExists` — "directory already has entry by that name" — from `Directory.AddChild` in [`boxo/mfs/dir.go`](https://github.com/ipfs/boxo/blob/main/mfs/dir.go)), so `ipfs files rm` is still required before every rewrite. Tellingly, the `files write` help itself describes `--to-files` as the way "to add a file *without modifying an existing one*".
- **No parent creation.** If the destination directory — or the target's parent — does not exist, the add fails with `MFS destination directory … does not exist`; `add` has no `--parents` equivalent, so `ipfs files mkdir --parents` is still required as well.
Counting commands and network round trips, `--to-files` is on par with the three-command sequence above (plus a temp-file detour because of the no-stdin limitation), and it is equally non-atomic between the steps. On a slow or flaky connection it is at best an equal, not an improvement: there is still no single, atomic "stream these bytes to that path, replacing whatever is there" operation, which is exactly the job `files write` is supposed to do.
## Related
- #8504 — the original UX-gap issue about `add` vs `files write` CID divergence; resolved via `add --to-files`, which (see Workaround) takes no stdin, cannot overwrite, and cannot create parents — so the gap for `files write` itself remains
- [v0.40 changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.40.md) — `Import.*` options introduced (incl. `Import.UnixFSDAGLayout` for `ipfs add`); `files write` raw-leaf fix for CIDv1 directories
- [v0.43 changelog](https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.43.md) — MFS writes now hold the same lock as `ipfs add` to protect in-flight writes from garbage collection, fixing a class of write-vs-GC corruption; the observations above are from v0.43.0
- [IPIP-499: UnixFS CID Profiles](https://specs.ipfs.tech/ipips/ipip-0499/) — reproducible CID generation across IPFS implementations
- [`boxo/ipld/unixfs/mod/dagmodifier.go`](https://github.com/ipfs/boxo/blob/main/ipld/unixfs/mod/dagmodifier.go) — hardcoded `trickle.Append` in the MFS write path
## Environment
- Kubo v0.43.0, linux-amd64
- Default configuration, no `Import.*` overrides, daemon running
Contributor guide
Research direction
Start by reproducing the repeated `ipfs files write --create --truncate` sequence and compare it with `ipfs add`. Read `boxo/ipld/unixfs/mod/dagmodifier.go`, `core/commands/add.go`, and `boxo/mfs/dir.go` to trace layout, chunking, and MFS replacement. Done means deterministic add-compatible CIDs, no zero-length blocks, and atomic overwrite behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, backend, cli
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100