common-workflow-language / common-workflow-language/cwltool

CWL fails to reconginze the existing files to generate new outputs using inplace editing

Open
#993 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

I am using CWL to generate some output based on certain existing output files in outdir.

---

## Expected Behavior

The CWL should take in the existing output files in the designated output directory and generate new outputs. The designated output dir consists of some .pickle files. The CWL should take them and generate single merged hdf5 file.

## Actual Behavior
Instead CWL scripts starts from the scratch. Which is processing wise time consuming

## Workflow Code
```
cwlVersion: v1.0
class: CommandLineTool
doc: Spladder

baseCommand: [python,-m,spladder.spladder]

hints:
cwltool:InplaceUpdateRequirement:
inplaceUpdate: true

requirements:
EnvVarRequirement:
envDef:
PYTHONPATH: /cluster/home/aalva/spladder/python
InlineJavascriptRequirement: {}
InitialWorkDirRequirement:
listing:
- entry: $(inputs.spladder_dir)
writable: true

inputs:
spladder_gtf:
type: File
inputBinding:
position: 3
prefix: -a
spladder_bams:
type: File[]
inputBinding:
position: 1
prefix: -b
itemSeparator: ","
secondaryFiles: .bai
spladder_dir:
type: Directory
inputBinding:
position: 2
prefix: -o
spladder_phase2:
type: string
inputBinding:
position: 4
prefix: -T
spladder_merge_graphs:
type: string
inputBinding:
position: 5
prefix: -M
spladder_qmode:
type: string
inputBinding:
position: 7
prefix: --qmode
spladder_quantify_graph:
type: string
inputBinding:
position: 6
prefix: --quantify_graph

outputs:
spladder_out:
type: Directory
outputBinding:
glob: $(inputs.spladder_dir.basename)

$namespaces:
cwltool: http://commonwl.org/cwltool#
```

## Full Traceback
```
Strted to run from scratch in temp file
```

## Your Environment
* cwltool version: v1.0

I ran it as `cwltool --enable-ext spl.cwl sp.yml`

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the provided CWL workflow, especially cwltool:InplaceUpdateRequirement and InitialWorkDirRequirement, then reproduce it with `cwltool --enable-ext spl.cwl sp.yml` using the reported cwltool v1.0 environment. Determine why the existing `.pickle` files in the designated output directory are not reused; done means the workflow recognizes them and produces the merged HDF5 output without restarting from scratch.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.