galaxyproject / galaxyproject/brc-analytics
Additional review findings from #1396 (dbt correctness, robustness, CI/reproducibility)
- Dominant language
- TypeScript
- Stars
- 7
- Forks
- 11
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 15
Description
Additional findings from a review of #1396 that aren't yet covered by the existing follow-ups (#1431 common names, #1433 dbt test detail, #1434 merged.dmp, #1435 taxdump chunking, #1436 build.py small fixes). Grouped by theme; none are merge-blockers.
## dbt model correctness
- **`not_null` test on species columns fails for above-species taxa** [confirmed]
`schema.yml` asserts `not_null` on `taxonomic_level_species` / `taxonomic_level_species_id`, but `taxonomy_assemblies.sql:6-7` computes them via `first(l.taxon_name) filter (l.rank = 'species')`, which is NULL when no lineage node has rank exactly `species` (a taxon registered at genus/`no rank`, or a viral taxon classified only above species). The test then fails (currently `warn`), and downstream consumers get silent NULLs where a species is assumed. Either relax the test or make the expectation explicit.
- **Nondeterministic name/id pairing in the taxonomic-level fields** [confirmed]
`taxonomy_assemblies.sql:6-7` uses two independent `first(...) filter (l.rank = 'X')` aggregates (one for `taxon_name`, one for `tax_id`) with no `ORDER BY`. DuckDB doesn't guarantee which row `first()` returns, so if a lineage has two nodes of the same configured rank the name and id can come from different nodes, and results can vary between rebuilds. (The `2d2fd91` "order names" fix addressed name-class ordering for common names, not these per-rank picks.) Add a deterministic `ORDER BY` / dedup so name and id come from the same node.
- **QC report labels skipped/warn dbt tests as failing** [plausible]
`transform/transform.py:29` sets `failed = status != "pass"`, so any dbt status other than `pass` — notably `skipped` (e.g. a test skipped because an upstream node errored) or `warn` — is reported under "Failing dbt tests" in the QC report, obscuring the real failure. Check for the failing states explicitly. (Adjacent to #1433 but a distinct correctness point.)
## Robustness
- **`.dmp` files opened without an explicit encoding** [confirmed]
`load/ncbi_taxonomy.py:38` `dmp_rows` uses `open(path)` with no `encoding=`. `names.dmp` contains non-ASCII taxon names; under a C/POSIX locale (`LANG` unset — common in minimal Docker/CI images) Python <3.15 defaults `open` to ASCII and raises `UnicodeDecodeError` on the first non-ASCII line, aborting the build. Use `open(path, encoding="utf-8")`.
- **`read_outbreaks` crashes on an empty/malformed outbreaks YAML** [plausible]
`build.py:220` does `pd.DataFrame(yaml.safe_load(stream)["outbreaks"])`. An empty (or fully commented-out) file makes `safe_load` return `None` → `None["outbreaks"]` `TypeError`; a file missing the top-level `outbreaks:` key → `KeyError`. The old `get_outbreak_taxonomy_ids` guarded this with `if not outbreaks_data: return []`. Restore a guard for the empty/keyless case. (Fits alongside #1436.)
- **Taxdump MD5 is fetched after the archive is downloaded** [plausible]
`load/ncbi_taxonomy.py` orchestration (~line 174) calls `download_taxdump` → `fetch_taxdump_md5` → `verify_taxdump_md5`. NCBI regenerates `taxdump.tar.gz` and its `.md5` periodically (and gzip output isn't byte-stable), so if a regeneration lands between the archive download and the MD5 fetch, the digests won't match and the build aborts with a spurious "MD5 does not match" even though nothing is corrupt. There's no retry. Consider fetching the MD5 first, or retrying the pair on mismatch. (Also note: extraction currently happens before verification, and the MD5 is same-origin so it's a corruption check, not tamper protection — low severity given `filter="data"` + member allowlist.)
## CI / reproducibility
- **The new dlt/dbt from-ncbi pipeline is never exercised in PR CI**
`run-checks.yml` only lints/format-checks Python; `api-tests` runs the TypeScript `build-brc-db`, not `build-*-from-ncbi`. A change that breaks a dbt model, the taxdump download/parse, or the DuckDB round-trip merges green and only surfaces in the weekly `update-catalog.yml` (BRC only). **`build-ga2-from-ncbi` is referenced by no workflow at all**, so the GA2 taxonomy build can silently rot. Consider a lightweight CI job that runs at least one from-ncbi build (or a scoped subset) for both catalogs.
- **The two `requirements.txt` dbt/dlt pins have already diverged**
`catalog/build/py/requirements.txt` pins `dbt-core==1.11.11` / `dbt-adapters==1.24.2` while `catalog/ga2/build/py/requirements.txt` pins `dbt-core==1.11.12` / `dbt-adapters==1.24.4` (plus other deltas like `humanize`), even though both run the identical shared dbt project. A behavior change in a dbt patch release can make the same taxonomy SQL produce different output for GA2 vs BRC — non-reproducible and hard to diagnose. Keep the shared engine pinned in lockstep across both files.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.