JacobLinCool / JacobLinCool/particeps

Materialize typed Parquet datasets with provenance

Open
#14 0 comments 0 reactions 0 assignees View on GitHub
area: analysis P1
Dominant language
Kotlin
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

> **Renamed by #23.** This issue was written while the project was called Android Data
> Collector. Its terminology has been updated to the Particeps identity now in force. The
> old spellings are rejected inputs, not an earlier dialect.

## Summary

Materialize the validated event stream from `particeps-analysis` as deterministic, strictly typed Parquet datasets with complete source provenance.

This issue implements Parquet only. It defines a narrow `DatasetSink` extension point for later work but does not add Postgres, D1, dashboards, or any other database connector.

## Requirements

### Strict schema materialization

- Consume only the validated/reassembled stream from #13.
- Resolve every collector, payload type, field, type, unit, and nullability rule through the registry in #10.
- Convert Protocol v1 decimal-string 64-bit values to lossless Arrow/Parquet integer types after strict bounds checking.
- Parse enumerations, booleans, integers, fixed binary, and strings according to the declared schema; do not infer types from observed values.
- Reject/quarantine unknown collector or payload schemas, undeclared fields, invalid enums, lossy values, unit mismatches, and nullability violations. Do not emit a generic map/JSON fallback column.
- Preserve wall time, continuous/monotonic time, time-zone context, boot-session identity, sequence numbers, and producer metadata at full declared precision.

### Dataset layout and provenance

- Partition safely and deterministically by experiment, configuration, collector, schema, and payload type, with path-safe encodings that cannot traverse the output root.
- Within files, use stable ordering and a documented row-group/file sizing policy.
- Make each row or row group traceable to:
- source ciphertext SHA-256;
- bundle ID;
- configuration digest;
- parser/catalog version;
- source object/path;
- identical-duplicate source set where applicable.
- Keep sensitive identifiers inside the protected dataset and out of filenames unless the documented data-handling policy explicitly permits them.
- Emit a dataset manifest containing schema fingerprints, partitions, row/file counts, source inventory digest, tool versions, conflicts/quarantines, and reproducibility metadata.
- Define a typed `DatasetSink` interface around validated records, but implement only the Parquet sink in this issue.

### Quality summaries

Produce offline, evidence-bounded summaries for:

- sequence gaps, overlaps, identical duplicates, and conflicts;
- boot sessions and incomparable monotonic intervals;
- wall-clock and time-zone changes;
- requested versus achieved sampling rates where the configuration and data support the calculation;
- collector start/stop/access/quota states;
- intervention/survey occurrence lifecycle.

Do not infer participant awareness, notification visibility, travel, presence, orientation, activity, or other behavior the recorded events cannot prove.

## Acceptance criteria

- Every shipped registry schema has deterministic Arrow/Parquet schema fixtures.
- Unknown schemas or invalid typed values fail closed without a generic fallback partition.
- Full-range supported 64-bit sequence/time values round-trip without floating-point conversion or precision loss.
- Identical validated inputs, catalog version, and options produce identical logical schemas, manifests, partition paths, ordering, and quality summaries.
- Any output row can be traced to its source ciphertext, bundle, configuration, parser/catalog version, and source object.
- Local and R2-derived validated streams produce equivalent Parquet datasets.
- Conflicts and quarantined bundles are reported and never silently materialized as authoritative rows.
- The shipped package has no database client or D1/Postgres-specific implementation.

## Documentation

Document Parquet schemas and partitioning, provenance columns/sidecars, manifest format, quality-metric definitions and limits, secure output handling, reproducibility workflow, and the future `DatasetSink` contract.

## Dependencies

Blocked by #10 (schema registry) and #13 (validated reassembly stream).

Contributor guide

Open the contributing guide

Research direction

Review blockers #10 and #13 first, then trace the validated/reassembled stream and schema registry entry points they provide. Define the DatasetSink boundary and Parquet-only implementation against the acceptance criteria; done means deterministic typed datasets, provenance and manifest output, bounded quality summaries, safe quarantine of invalid inputs, and no database connector.

Written by the indexing model from the issue text.

Assessment

Tech stack
kotlin
Domain
data-engineering, documentation
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.