JacobLinCool / JacobLinCool/particeps
Build offline bundle inventory, verification, and reassembly tooling
- Dominant language
- Kotlin
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
> **Renamed by #23.** This issue was written while the project was called Android Data
> Collector. Its terminology has been updated to the Particeps identity now in force. The
> old spellings are rejected inputs, not an earlier dialect.
## Summary
Create a standalone Python `particeps-analysis` package and CLI that inventories encrypted bundles, verifies and decrypts them offline, and produces one validated event stream for downstream materialization.
The receiver remains ciphertext-only. This tool reads either local `.partexp` files or objects obtained from an S3-compatible/R2 source, and the researcher supplies decryption keys locally.
## Requirements
### Source and inventory model
- Define a narrow `BundleSource` interface with:
- a local file/directory implementation; and
- an S3-compatible implementation suitable for Cloudflare R2.
- Separate ciphertext acquisition/inventory from decryption. A source sync must be able to finish and emit an immutable inventory before private keys are loaded.
- Record source object identity, object/custom metadata when present, byte count, transport/object SHA-256, discovery time, and local cache identity.
- Make repeated inventory/sync idempotent; never silently replace a cached object whose identity now has different bytes.
- Support both automatic-upload objects and participant-provided manual `.partexp` files without adding functionality to the receiver.
### Verification and authenticated decryption
For every candidate bundle, verify in a fail-closed order:
- local/object byte count and SHA-256;
- receipt/custom metadata when available;
- Protocol v1 framing and bounds;
- HPKE recipient/context and AES-GCM authentication;
- canonical embedded configuration bytes;
- Ed25519 signature and signer fingerprint;
- configuration SHA-256;
- outer/inner experiment, configuration, participant-instance, bundle, producer, and format identities;
- claimed versus authenticated actual sequence range and event count;
- per-event schema identity, bounds, and monotonic/global sequencing rules.
Plaintext may exist only in a mode-0600 staging area. Publish it to the validated intermediate representation only after complete AEAD, structure, identity, range, and schema validation. If any check fails, quarantine the entire bundle with a bounded reason code and produce no partial rows.
### Reassembly and data-quality semantics
- Partition reassembly by `(experiment_id, configuration_id)` before considering participant streams.
- Within a participant stream, de-duplicate by `(participant_instance_id, sequence_number)`.
- If duplicate keys have byte-/value-identical authenticated events, retain one and record identical-source provenance.
- If duplicate keys contain different authenticated content, emit an explicit conflict and publish neither as an unquestioned winner. Do not use last-write-wins.
- Classify uploaded range overlap, identical duplicate, conflicting duplicate, not-yet-received suffix, explicitly reclaimed prefix, and proven gap separately.
- Do not label absence as a true gap unless available authenticated lifecycle/range evidence makes that conclusion possible.
- Preserve boot-session and wall/continuous-time context without trying to order incomparable monotonic values across boots.
- Produce deterministic machine-readable inventory, validation, quarantine, reassembly, conflict, and quality reports.
### Package and CLI
- Provide typed Python APIs plus stable CLI commands for inventory/sync, verify, decrypt/reassemble, inspect, and report.
- Keep credentials and private keys out of command histories, process listings, logs, reports, and exception text.
- Pin parsing/crypto dependencies and include hostile corpus/fuzz/property tests using #8's shared fixtures.
- Do not add an online decrypt service, receiver callback, database writer, or schema-inference fallback.
## Acceptance criteria
- Local and R2 sources containing identical objects yield the same validated stream and provenance.
- Wrong keys, wrong configuration, wrong context, corruption, truncation, metadata mismatch, malformed framing, unknown schema, and range mismatch quarantine the whole bundle without partial output.
- Identical duplicates collapse deterministically; conflicting duplicates remain explicit conflicts and never silently overwrite one another.
- Tests distinguish overlap, conflict, pending suffix, reclaimed prefix, and proven gap.
- Plaintext staging has mode 0600, is not published before full validation, and is cleaned or retained under an explicit secure quarantine policy after failure.
- Re-running the same inventory and reassembly produces byte-stable reports.
- Logs and reports do not reveal private keys, credentials, assigned participant IDs, or decrypted payload values unless an explicit inspection command requests the latter.
## Documentation
Document installation, key handling, local and R2 source setup, inventory workflow, verification order, quarantine policy, reassembly semantics, privacy-safe diagnostics, and reproducible examples.
## Dependencies
Blocked by #8 and #9. The local-file path can be implemented once those contracts are final; the S3-compatible/R2 source integration additionally depends on #12.
Contributor guide
Research direction
Start by reading the contracts in #8 and #9 and their shared fixtures; the local-file path depends on those contracts, while the S3-compatible/R2 path also depends on #12. Work through the particeps-analysis CLI entry points: inventory/sync, verify, decrypt/reassemble, inspect, and report. Done means the listed acceptance cases pass with deterministic reports, no partial output, and safe handling of plaintext and credentials.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli, data, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100