nf-core / nf-core/atacseq

Docker: deprecate docker.userEmulation and use docker.runOptions; mitigates Trim Galore pipe freeze

Open
#435 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Nextflow
Stars
229
Forks
141
Avg merge
13h 41m
Merged PRs (30d)
1

Description

Description of the bug

Summary

When running the pipeline with -profile docker, two related issues occur:

  1. Deprecated config: Nextflow logs a warning that docker.userEmulation is no longer supported and should be removed from the config.
  2. Trim Galore can freeze: The Trim Galore step sometimes hangs in the cutadapt stage when run inside Docker. This is a known upstream issue (Trim Galore #207) caused by small pipe buffers in container environments.

Root cause (Trim Galore freeze)

  • By default, Docker containers run as root (UID 0). The Linux kernel accounts pipe buffer usage per UID.
  • When the total pipe usage for root exceeds pipe-user-pages-soft, new pipes are limited to 2 pages (8192 bytes) instead of the default 16 pages (~64 KB).
  • With many Nextflow tasks (each creating pipes, e.g. cutadapt ↔ trim_galore), root’s quota is easily exceeded, so new pipes get the small size and Trim Galore can block waiting for the last few lines from cutadapt (see Trim Galore #207).
  • Running containers as the host user (non-root) spreads pipe usage by UID and often avoids hitting that limit, which can mitigate or avoid the freeze.

Current config (problematic)

In nextflow.config, the docker profile uses the deprecated option:

docker {
docker.enabled = true
docker.userEmulation = true // deprecated, no longer supported by Nextflow
...
}

Suggested change

Remove docker.userEmulation = true from the docker profile (to clear the warning and stop relying on removed behaviour).
Add docker.runOptions = '-u $(id -u):$(id -g)' to the docker profile so containers run as the current user.

Command used and terminal output
nextflow -bg -log test.log -q run ~/pipeline/nf-core/atacseq-2.1.2 -profile docker -params-file params.json -resume
Relevant files

No response

System information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Open nextflow.config and inspect the docker profile. Remove docker.userEmulation and add the suggested docker.runOptions value, then run the provided Nextflow command to confirm the deprecation warning is gone and the Docker tasks use the current user.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker
Domain
devops
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.