ngless-toolkit / ngless-toolkit/ngless

Introduce computation stages - validation, download, prepare, ...

Open
#135 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement performance
Dominant language
Rust
Stars
152
Forks
23
PR merge metrics
No merged PRs in 30d

Description

Currently NGLess uses two stages for execution. A first stage verifies that the script and output files are consistent (equivalent to --validate-only) and a second stage where computation happens if the first stage finishes successfully.

However, the current implementation performs downloads, indexing and computation during the same (second) stage.
If using the parallel module, this can lead to jobs waiting on each other for significant amounts of time. This happens during indexing and initialization of internal and external modules, as well as, during downloads, leading to failures or delays due to connectivity problems or slow networks speeds.

For example, mapping to hg19 only downloads and indexes the files when the map() step is reached for the first time.

This limitation often leads to workflows that follow a run one sample first and if it finishes run all others approach.

If implementing a staged execution, an ngless workflow could look like:

# (run once) Ensure ngless is correctly installed
ngless --check-install

# (run once) Check that the script is valid and inputs/outputs are as expected
ngless --validate-only script.ngl

# (run once/multiple) Download and index all dependencies (references, resources from internal modules, initialization of external modules, indexing, etc...)
ngless --ensure-dependencies script.ngl

# (run once/multiple) Interpret the script, possibly in parallel
ngless script.ngl 

An advantage of --ensure-dependencies is that resources could be downloaded, indexed, ... in parallel, something which currently happens sequentially.

Additionally, execution of script.ngl would have predictable behavior for a user regardless of being the first time the command is being executed.

This issue is also in line with #71 which proposes a setup phase for external modules. Such phase would also run during --ensure-dependencies.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how NGLess currently handles the validation-only stage and the computation stage, then compare the proposed commands with the setup phase described in issue #71. The work is complete when dependency downloads, indexing, and module initialization can be performed through --ensure-dependencies and execution has predictable behavior afterward.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.