galaxyproject / galaxyproject/brc-analytics

Coordinated ingestion and internal consistency across related data

Open
#1,049 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
7
Forks
11
Avg merge
2d 12h
Merged PRs (30d)
16

Description

## Description

BRC Analytics currently ingests related data via multiple scripts and upstream sources (e.g. organism and assembly metadata from NCBI; SRA-derived summaries from ENA) and potentially at different times.

As ingestion expands (e.g. adding SRA metadata), this raises concerns about internal consistency: related entities may reflect different upstream states even when sourced from the same resource.

## Core concerns

- Organisms/ assemblies and derived summaries (e.g. SRA run counts) are generated by different scripts.
- Different scripts can potentially be run at different times, or using different resources (ENA and NCBI take time to sync)
- It is unclear yet how ingesting SRA metadata will work, but it does seem clear that the data ingested will relate to data produced by these existing scripts
- Relationships between all of these entities are implicit and at risk of internal inconsistency

## Impact/ Urgency

This is another one that isn't really 'critical' yet but is probably worth thinking some about before we make it any worse. It seems to me we're at a sort of inflection point with the SRA metadata discussions. In the immediate term (pre-SRA-metadata) I think our worst case is stepper UI possibly claiming the wrong number of sequences to browse from ENA.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by mapping the existing ingestion scripts and their NCBI and ENA inputs, including how organism, assembly, and SRA-derived data are related. Determine what coordinated ingestion approach would keep related entities consistent across upstream updates; the issue does not yet define a specific implementation or test target.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.