AllenInstitute / AllenInstitute/ConnectsCommonConnectivity

CellCellConnectivityLong: add connectome_id

Open
#17 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement schema
Dominant language
Jupyter Notebook
Stars
4
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Problem: no measurement-context discriminator — rows from two measurement contexts are indistinguishable and their overwrite predicates clobber each other. Hence today's one-subdirectory-per-connectome workaround (cellcellconnectivitylong_*).

Semantics: connectome_id identifies the measurement context (segmentation version + proofreading state + measurement semantics), not the cohort. Cohorts are read-side filters: write the superset connectome once; pull subsets by joining dataitem_dataset_association or cluster membership. Minnie's two folders (proofread_pre_to_csm_post, proofread_to_proofread) are two measurement contexts → two connectome_ids in one table.

Fix: add connectome_id; register a WriteSpec (overwrite_scoped on (project_id, connectome_id)); rerun affected ETLs with the new io + schema (no folder migration); add a subset-read helper (ties to #23). Needs its own planning session before implementation.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the CellCellConnectivityLong schema, its io and WriteSpec registration, and the affected ETLs. Review the planned relationship to issue #23 and use the planning session to settle the subset-read helper. Done means the new connectome_id supports scoped overwrites, affected ETLs use the updated schema and io, and no folder migration is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.