matx64 / matx64/synche

Protocol & architecture follow-ups: block-level transfer, atomicity, rename modelling

Open
#34 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Tracking issue for design-level improvements to the versioning / sync subsystem that become tractable after #32 (security hardening) and #33 (version-vector correctness) land. None of these are urgent on their own; they are filed together so we can prioritise as a group when the fundamentals are stable.

Refs refreshed against main @ a8c2c9a (post #32–#44). Line numbers below reflect the current tree.


[PERF / DESIGN] Full hashes, full handshakes, full-file transfers
  • Startup / scan hashes every file as a whole-file SHA-256 (EntryManager::walk_dir, app/src/application/state/entry_manager.rs:99, calling compute_hash at app/src/utils/fs.rs:47).
  • Every handshake re-ships the full entry map for every peer (EntryManager::get_handshake_data, app/src/application/state/entry_manager.rs:895).
  • Every transfer streams the entire file (app/src/infra/network/tcp/{sender,receiver}.rs).

Syncthing's Block Exchange Protocol is built around per-file version vectors plus block hashes and block requests, which enables incremental transfer, resumable sync, and bounded request tracking. See https://docs.syncthing.net/specs/bep-v1.html.

Suggested direction: evaluate block-level chunking (128 KiB blocks, per-block SHA-256) plus a sequence-number-based index-exchange model (Syncthing's ClusterConfig / Index / IndexUpdate). Significant change — worth a separate design doc before implementation. Overlaps with #8.

[LOW] Sender zero-pads on file-shrink mid-transfer

TcpSender::stream_file_to (app/src/infra/network/tcp/sender.rs:157) pads short reads to the advertised size to keep wire framing consistent (:181); the receiver then rejects on hash mismatch (app/src/infra/network/tcp/receiver.rs:167-182). Correct in the sense that no bad data lands on disk, but wasteful — we ship up to entry_size zero bytes before the receiver discovers the mismatch.

Suggested direction: on short read, abort the send and drop the connection. The receiver fails cleanly via framing rather than after a full padded transfer.

[LOW] Cross-device finalise leftover-temp hygiene

Partially addressed by #33 B1. The staging→home_path move moved out of the TCP receiver and into EntryManager::commit_staged_transfer (app/src/application/state/entry_manager.rs:644). The cross-device branch (CrossesDevices, :744) now copies to a temp sibling inside the target directory and then renames that sibling onto the target (:759) — it no longer copy + remove_files directly over the user file, so the user file is never left half-written. The staged bytes themselves are an RAII StagedTransfer whose Drop cleans the staging temp dir.

Remaining hygiene gap: a crash mid-copy can still leak a partial temp sibling in the target dir (or a staging dir if the process dies before Drop runs).

Suggested direction: add a startup sweep of leftover synche-* temp/staging artifacts older than X hours; optionally stage on the same filesystem as home_path so the cross-device branch is never hit.

[LOW] Rename is modelled as delete + create, not as a move

There is no Move variant in TransportData (app/src/domain/transport.rs:95). notify rename events become a tombstone for the old path plus a fresh version: {local_id: 0} for the new path. Two peers concurrently renaming the same file to different names get an unrecoverable both-renames-win outcome (file exists under both names).

Suggested direction: detect rename events from notify and ship them as a single Move carrying the source entry's full version vector. The conflict-resolution path for concurrent renames needs to be designed alongside.

[LOW] EntryInfo::compare allocates on every call

EntryInfo::compare (app/src/domain/entry/info.rs:37) builds a HashSet<Uuid> of the union of both vectors on every call. Cheap, but called on every Metadata, Request, and handshake entry. Could iterate the two maps directly with merge-bookkeeping for a trivial allocation-free implementation.

[LOW] Directories carry version vectors that are never meaningfully compared

Directory entries get a version: HashMap<...> populated in EntryManager::walk_dir (app/src/application/state/entry_manager.rs:99), but the directory case is always short-circuited to Equal by the kind == kind && hash == hash check at the top of compare (app/src/domain/entry/info.rs:37) — directories have hash: None, which is always equal between two dirs. Worth a comment in EntryInfo so future readers don't assume parity with files, or drop the field for directories entirely.


Depends on: #32, #33.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin by reading the dependency issues #32 and #33, then inspect EntryManager in app/src/application/state/entry_manager.rs, fs.rs, the TCP sender and receiver, and the transport and EntryInfo types named in the issue. This tracking issue needs to be split into separately scoped design or implementation tasks; each is done only when its design, affected paths, and validation criteria are defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.