AllenInstitute / AllenInstitute/ConnectsCommonConnectivity
Add merge_scoped write mode with declared merge_on keys
Nobody has claimed this yet.
- Dominant language
- Jupyter Notebook
- Stars
- 4
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Problem: only overwrite_scoped and append_new_by_id exist — no upsert. Incremental workflows (adding dataitems, revising feature values, QC arriving over time) silently lose data (#15) or no-op.
Fix: add write_mode="merge_scoped" + merge_on: list[str] to WriteSpec; implement with delta-rs MERGE; dedupe within batch; assign modes per class per the table in planning/20260820/2026-08-20_work_packages.md §P4. Replaces append_new_by_id for DataItem (fixes silent no-op metadata updates).
Scope: merge is for identity-bearing metadata/association tables (10³–10⁶ rows). Bulk long tables and wide matrices stay overwrite_scoped — MERGE joins incoming vs existing rows, wasteful at 10⁷.
Decide in the PR: pure upsert (accumulate) vs sync-with-delete within a sub-scope. Pure upsert fits our workflows; deletion stays explicit (#21).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with WriteSpec and planning/20260820/2026-08-20_work_packages.md §P4, then trace the existing overwrite_scoped and append_new_by_id modes. Determine how merge_scoped, merge_on, batch deduplication, and delta-rs MERGE should fit together, and resolve pure upsert versus sync-with-delete in the PR. Done means the modes are assigned per the table and metadata updates no longer silently no-op.
Written by the indexing model from the issue text.
Assessment
- Domain
- data, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100