common-workflow-language / common-workflow-language/cwltool
intermediate outputs directories persist too long
- Dominant language
- Python
- Stars
- 376
- Forks
- 255
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 12
Description
From https://github.com/common-workflow-language/cwltool/pull/898#issuecomment-431154202 by @WESClarke
> For me this still leaves intermediate directories using the example in [#892](
> https://github.com/common-workflow-language/cwltool/issues/892).
>
> First issue: output_dirs is always empty at L104 in my tests
>
> [https://github.com/common-workflow-language/cwltool/blob/e4bc1021ef406aff98fe68b1d29edb78eaf72c95/cwltool/executors.py#L98-L104](https://github.com/common-workflow-language/cwltool/blob/e4bc1021ef406aff98fe68b1d29edb78eaf72c95/cwltool/executors.py#L98-L104)
>
> Second issue: job_dir is created using tmp_outdir_prefix but also does not get deleted. I tried setting L729 to `shutil.rmtree(job_dir, True)` without success.
>
> [https://github.com/common-workflow-language/cwltool/blob/e4bc1021ef406aff98fe68b1d29edb78eaf72c95/cwltool/job.py#L728-L729](https://github.com/common-workflow-language/cwltool/blob/e4bc1021ef406aff98fe68b1d29edb78eaf72c95/cwltool/job.py#L728-L729)
>
>
> My solutions are pretty naive but I am including them anyways.
>
> Issue 1: I have moved the creation of the job_dir outside of the call to `_job_popen` and then called for its removal with the other temporary directories.
>
> [https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/job.py#L294-L306](https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/job.py#L294-L306)
>
> [https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/job.py#L385-L389](https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/job.py#L385-L389)
>
> Issue 2: I have just used `output_dirs = self.output_dirs` for both cases, knowing that I must be missing something. However, I have tested this with and without --cachedir without issue.
>
> [https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/executors.py#L98-L105](https://github.com/WEClarke/cwltool/blob/042902254882d96109f93b5291452626ee6f83f8/cwltool/executors.py#L98-L105)
>
>
>
>
Contributor guide
Assessment
This issue has not been assessed yet.