VEuPathDB / VEuPathDB/ApiCommonWorkflow

Make genomics project dependency lists data-driven instead of hand-copied

Open
#107 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Perl
Stars
1
Forks
2
Avg merge
2d 17h
Merged PRs (30d)
11

Description

Problem

The list of genomic projects (PlasmoDB, ToxoDB, TriTrypDB, VectorBase, AmoebaDB, CryptoDB, FungiDB, PiroplasmaDB, GiardiaDB, MicrosporidiaDB, TrichDB, HostDB, and now HelminthDB) is hand-copied into 7 separate blocks across 2 files:

Main/lib/xml/workflowTemplates/genomicsRoot.xml — 5 blocks:

  • sampleDetailsForEda
  • OrthoMCLDependent
  • alphaFold
  • ExternalResourceUrl
  • analyzeWebreadyPartitionedTables

Main/lib/xml/workflow/orthomclGenomics.xml — 2 blocks:

  • checkOrthoCoreProteomeDirs dependsExternal list
  • checkOrthoPeripheralProteomeDirs dependsExternal list

The only thing keeping them in sync is a comment at genomicsRoot.xml:59:

ATTENTION: This graph has dependsExternals by PROJECT. If you add a project above, you must also add it here

A load-bearing sticky note is the tell that the structure is wrong. Adding HelminthDB (#106) meant 7 edits and a manual audit to confirm nothing was missed; a silently omitted entry produces a workflow graph that runs but quietly skips a project's data.

Proposed direction

The project list is data, not structure. Options, roughly in order of preference:

  1. datasetTemplate-driven dependency expansion — the genomicsProject datasetTemplate already enumerates projects to instantiate the per-project subgraphs. Extend ReFlow so a step can declare a dependency on all instances of a template (e.g. <dependsAll template="genomicsProject"/>, and for orthomclGenomics.xml something like <dependsExternalAll template="genomicsProject" step="checkOrthoCoreProteomeDirs"/>). Single source of truth, no duplication anywhere.

  2. Shared constant + expansion — declare the list once (<constant name="allGenomicsProjects">) and have the XML expand it into depends / dependsExternal entries. Less invasive than (1), still one place to edit.

  3. Validation-only fallback — if neither expansion mechanism is worth building, add a test/lint step that asserts all 7 blocks contain the same project set. Doesn't remove the duplication but converts a silent skip into a loud failure.

Option 1 also makes the naming convention (${projectName}_checkOrthoCoreProteomeDirs) explicit in the graph rather than implied by string concatenation in two places.

Acceptance

Adding project #14 should be a one-line change in one file, with no cross-file audit required.

🤖 Generated with Claude Code

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the seven dependency blocks in Main/lib/xml/workflowTemplates/genomicsRoot.xml and Main/lib/xml/workflow/orthomclGenomics.xml, then trace how ReFlow expands datasetTemplate dependencies. Compare the proposed template-driven and constant-based approaches, and inspect existing workflow tests or validation commands if present. Done means adding project #14 requires one line in one file without a cross-file audit.

Written by the indexing model from the issue text.

Assessment

Tech stack
perl, xml
Domain
build-system, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.