Protocol & architecture follow-ups: block-level transfer, atomicity, rename modelling
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Tracking issue for design-level improvements to the versioning / sync subsystem that become tractable after #32 (security hardening) and #33 (version-vector correctness) land. None of these are urgent on their own; they are filed together so we can prioritise as a group when the fundamentals are stable.
Refs refreshed against
main@ a8c2c9a (post #32–#44). Line numbers below reflect the current tree.
[PERF / DESIGN] Full hashes, full handshakes, full-file transfers
- Startup / scan hashes every file as a whole-file SHA-256 (
EntryManager::walk_dir,app/src/application/state/entry_manager.rs:99, callingcompute_hashatapp/src/utils/fs.rs:47). - Every handshake re-ships the full entry map for every peer (
EntryManager::get_handshake_data,app/src/application/state/entry_manager.rs:895). - Every transfer streams the entire file (
app/src/infra/network/tcp/{sender,receiver}.rs).
Syncthing's Block Exchange Protocol is built around per-file version vectors plus block hashes and block requests, which enables incremental transfer, resumable sync, and bounded request tracking. See https://docs.syncthing.net/specs/bep-v1.html.
Suggested direction: evaluate block-level chunking (128 KiB blocks, per-block SHA-256) plus a sequence-number-based index-exchange model (Syncthing's ClusterConfig / Index / IndexUpdate). Significant change — worth a separate design doc before implementation. Overlaps with #8.
[LOW] Sender zero-pads on file-shrink mid-transfer
TcpSender::stream_file_to (app/src/infra/network/tcp/sender.rs:157) pads short reads to the advertised size to keep wire framing consistent (:181); the receiver then rejects on hash mismatch (app/src/infra/network/tcp/receiver.rs:167-182). Correct in the sense that no bad data lands on disk, but wasteful — we ship up to entry_size zero bytes before the receiver discovers the mismatch.
Suggested direction: on short read, abort the send and drop the connection. The receiver fails cleanly via framing rather than after a full padded transfer.
[LOW] Cross-device finalise leftover-temp hygiene
Partially addressed by #33 B1. The staging→
home_pathmove moved out of the TCP receiver and intoEntryManager::commit_staged_transfer(app/src/application/state/entry_manager.rs:644). The cross-device branch (CrossesDevices,:744) now copies to a temp sibling inside the target directory and then renames that sibling onto the target (:759) — it no longercopy + remove_files directly over the user file, so the user file is never left half-written. The staged bytes themselves are an RAIIStagedTransferwhoseDropcleans the staging temp dir.
Remaining hygiene gap: a crash mid-copy can still leak a partial temp sibling in the target dir (or a staging dir if the process dies before Drop runs).
Suggested direction: add a startup sweep of leftover synche-* temp/staging artifacts older than X hours; optionally stage on the same filesystem as home_path so the cross-device branch is never hit.
[LOW] Rename is modelled as delete + create, not as a move
There is no Move variant in TransportData (app/src/domain/transport.rs:95). notify rename events become a tombstone for the old path plus a fresh version: {local_id: 0} for the new path. Two peers concurrently renaming the same file to different names get an unrecoverable both-renames-win outcome (file exists under both names).
Suggested direction: detect rename events from notify and ship them as a single Move carrying the source entry's full version vector. The conflict-resolution path for concurrent renames needs to be designed alongside.
[LOW] EntryInfo::compare allocates on every call
EntryInfo::compare (app/src/domain/entry/info.rs:37) builds a HashSet<Uuid> of the union of both vectors on every call. Cheap, but called on every Metadata, Request, and handshake entry. Could iterate the two maps directly with merge-bookkeeping for a trivial allocation-free implementation.
[LOW] Directories carry version vectors that are never meaningfully compared
Directory entries get a version: HashMap<...> populated in EntryManager::walk_dir (app/src/application/state/entry_manager.rs:99), but the directory case is always short-circuited to Equal by the kind == kind && hash == hash check at the top of compare (app/src/domain/entry/info.rs:37) — directories have hash: None, which is always equal between two dirs. Worth a comment in EntryInfo so future readers don't assume parity with files, or drop the field for directories entirely.
Depends on: #32, #33.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by reading the dependency issues #32 and #33, then inspect EntryManager in app/src/application/state/entry_manager.rs, fs.rs, the TCP sender and receiver, and the transport and EntryInfo types named in the issue. This tracking issue needs to be split into separately scoped design or implementation tasks; each is done only when its design, affected paths, and validation criteria are defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100