FilOzone / FilOzone/ipfs2foc

[Feature]: dedupe CIDs in a run and against what the account already stores

Open
#58 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
5d 21h
Merged PRs (30d)
1

Description

Storing the same bytes twice means paying twice. In a real 2.2M-CID inventory, 61,766 CIDs (~2.9%) appeared in two different spaces of the same account, so the overlap is routine rather than theoretical.

Two parts, with very different shapes.

## 1. Dedupe within a run

Mostly built. File intake canonicalizes each line through `toCanonicalCidV1` and dedupes on that form (`packages/core/src/cid-list.ts`), and the CLI inserts with `ON CONFLICT(cid) DO NOTHING` (`packages/cli/src/db.ts:236`).

The gap is the console's union. `cid-list.ts` deliberately returns the *input spelling*, and the `cids` memo in `app/src/app.tsx` dedupes the paste plus file union with `new Set` over raw strings. `Qm…X` pasted and `bafybei…X` loaded from a file are the same content, so they survive as two rows, become two pieces, and get billed twice. Dedupe the union on the canonical form, keyed the same way rows and saved results are.

## 2. Skip CIDs the account already stores

Design is open, and the obvious approach does not scale.

Chain is the canonical record: every piece we add carries its `ipfsRootCID` as piece metadata (`app/src/submit.ts:190`). But `getPieceMetadata(dataSetId, pieceId, key)` is a forward lookup, and there is no reverse index from a root CID back to a piece. Answering "does this account already store X?" by walking the account's pieces costs one contract read per piece. At millions of pieces that is millions of reads and RPC exhaustion. Not viable.

IPNI (`cid.contact`) is not a substitute on its own, for two reasons:

- It reports **announcement, not possession**. `--check-ipni` is documented as exactly that in `packages/cli/src/report.ts`; possession is proven by PDP, not by a routing record.
- It is **not account-scoped**. A record can come from the operator's old pinning service or any unrelated party. Treating that as "already stored" would skip a CID the account does not hold and is not paying for, and the data would silently never land. That is the failure this tool exists to prevent.

Options to weigh:

1. The run manifest (shipped in browser and CLI). Exact and free for runs this tool performed; useless if the operator lost it or migrated by other means. Couples to the open manifest question.
2. Store the manifest as a piece in the data set, giving the account its own reverse index: one retrieval, reconciled against chain.
3. IPNI as a cheap prefilter only, then confirm the candidates against the account's own data sets on chain. Bounds the reads to candidates, but inherits the false-positive risk above.
4. An indexer or subgraph over piece metadata.

## Done means

- A run never prepares the same content twice, whatever the CID spelling or which input carried it.
- Before preparing, an operator sees how many of their CIDs their account already stores, computed without a per-CID chain read.
- A skip is backed by evidence the account holds the piece, never by an announcement record.

Contributor guide

Open the contributing guide

Research direction

Start with packages/core/src/cid-list.ts, app/src/app.tsx, app/src/submit.ts, packages/cli/src/db.ts, and packages/cli/src/report.ts to trace canonicalization, unioning, insertion, metadata, and IPNI reporting. Compare the manifest, stored-manifest, IPNI-prefilter, and indexer options against the account-scope and read-scaling requirements. Done means one preparation per content and account-held counts backed by evidence rather than announcements.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.