nf-core patch does not work on remote repos that contain binary/compressed files
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 322
- Forks
- 255
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 5
Description
Description of the bug
Hello!
We have a remote nf-core modules repository that we use extensively in the Nextflow MGnify pipelines here: https://github.com/EBI-Metagenomics/nf-modules/tree/main
I've recently been making updates to one of the pipelines that uses the detect_rna subworkflow here: https://github.com/EBI-Metagenomics/nf-modules/tree/main/subworkflows/ebi-metagenomics/detect_rna/
I've made some small local changes in said pipeline and wanted to run make a diff using patch, like so:
nf-core subworkflows --git-remote https://www.github.com/ebi-metagenomics/nf-modules patch detect_rna
However, I get this error:
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/chrisata/miniforge3/envs/nextflow/bin/nf-core:10 in <module> │
│ │
│ 7 │
│ 8 if __name__ == '__main__': │
│ 9 │ sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) │
│ ❱ 10 │ sys.exit(run_nf_core()) │
│ 11 │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/nf_core/__main__.py:183 in │
│ run_nf_core │
│ │
│ 180 │ │ │ log.debug(f"Could not check latest version: {e}") │
│ 181 │ │ stderr.print("\n") │
│ 182 │ # Launch the click cli │
│ ❱ 183 │ nf_core_cli(auto_envvar_prefix="NFCORE") │
│ 184 │
│ 185 │
│ 186 @tui( │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/rich_click/rich_command.py │
│ :404 in __call__ │
│ │
│ 401 │ │ # Include this here because I run into a false warning │
│ 402 │ │ # in the PyCharm IDE otherwise; for some reason PyCharm doesn't │
│ 403 │ │ # seem to think RichGroups are callable. (No issues with Mypy, though.) │
│ ❱ 404 │ │ return super().__call__(*args, **kwargs) │
│ 405 │
│ 406 │
│ 407 class RichCommandCollection(CommandCollection, RichGroup): │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/core.py:1462 in │
│ __call__ │
│ │
│ 1459 │ │
│ 1460 │ def __call__(self, *args: t.Any, **kwargs: t.Any) -> t.Any: │
│ 1461 │ │ """Alias for :meth:`main`.""" │
│ ❱ 1462 │ │ return self.main(*args, **kwargs) │
│ 1463 │
│ 1464 │
│ 1465 class _FakeSubclassCheck(type): │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/rich_click/rich_command.py │
│ :187 in main │
│ │
│ 184 │ │ try: │
│ 185 │ │ │ try: │
│ 186 │ │ │ │ with self.make_context(prog_name, args, **extra) as ctx: │
│ ❱ 187 │ │ │ │ │ rv = self.invoke(ctx) │
│ 188 │ │ │ │ │ if not standalone_mode: │
│ 189 │ │ │ │ │ │ return rv │
│ 190 │ │ │ │ │ # it's not safe to `ctx.exit(rv)` here! │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/core.py:1850 in │
│ invoke │
│ │
│ 1847 │ │ │ │ super().invoke(ctx) │
│ 1848 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1849 │ │ │ │ with sub_ctx: │
│ ❱ 1850 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1851 │ │ │
│ 1852 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1853 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/core.py:1850 in │
│ invoke │
│ │
│ 1847 │ │ │ │ super().invoke(ctx) │
│ 1848 │ │ │ │ sub_ctx = cmd.make_context(cmd_name, args, parent=ctx) │
│ 1849 │ │ │ │ with sub_ctx: │
│ ❱ 1850 │ │ │ │ │ return _process_result(sub_ctx.command.invoke(sub_ctx)) │
│ 1851 │ │ │
│ 1852 │ │ # In chain mode we create the contexts step by step, but after the │
│ 1853 │ │ # base command has been invoked. Because at that point we do not │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/core.py:1246 in │
│ invoke │
│ │
│ 1243 │ │ │ echo(style(message, fg="red"), err=True) │
│ 1244 │ │ │
│ 1245 │ │ if self.callback is not None: │
│ ❱ 1246 │ │ │ return ctx.invoke(self.callback, **ctx.params) │
│ 1247 │ │
│ 1248 │ def shell_complete(self, ctx: Context, incomplete: str) -> list[CompletionItem]: │
│ 1249 │ │ """Return a list of completions for the incomplete value. Looks │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/core.py:814 in │
│ invoke │
│ │
│ 811 │ │ │
│ 812 │ │ with augment_usage_errors(self): │
│ 813 │ │ │ with ctx: │
│ ❱ 814 │ │ │ │ return callback(*args, **kwargs) │
│ 815 │ │
│ 816 │ def forward(self, cmd: Command, /, *args: t.Any, **kwargs: t.Any) -> t.Any: │
│ 817 │ │ """Similar to :meth:`invoke` but fills in default keyword │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/click/decorators.py:34 in │
│ new_func │
│ │
│ 31 │ """ │
│ 32 │ │
│ 33 │ def new_func(*args: P.args, **kwargs: P.kwargs) -> R: │
│ ❱ 34 │ │ return f(get_current_context(), *args, **kwargs) │
│ 35 │ │
│ 36 │ return update_wrapper(new_func, f) │
│ 37 │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/nf_core/__main__.py:1641 │
│ in subworkflows_patch │
│ │
│ 1638 │ │ if remove: │
│ 1639 │ │ │ subworkflow_patch.remove(tool) │
│ 1640 │ │ else: │
│ ❱ 1641 │ │ │ subworkflow_patch.patch(tool) │
│ 1642 │ except (UserWarning, LookupError) as e: │
│ 1643 │ │ log.error(e) │
│ 1644 │ │ sys.exit(1) │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/nf_core/components/patch.p │
│ y:117 in patch │
│ │
│ 114 │ │ # Write the patch to a temporary location (otherwise it is printed to the screen │
│ 115 │ │ patch_temp_path = tempfile.mktemp() │
│ 116 │ │ try: │
│ ❱ 117 │ │ │ ComponentsDiffer.write_diff_file( │
│ 118 │ │ │ │ patch_temp_path, │
│ 119 │ │ │ │ component, │
│ 120 │ │ │ │ self.modules_repo.repo_path, │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/nf_core/components/compone │
│ nts_differ.py:167 in write_diff_file │
│ │
│ 164 │ │ if dsp_to_dir is None: │
│ 165 │ │ │ dsp_to_dir = to_dir │
│ 166 │ │ │
│ ❱ 167 │ │ diffs = ComponentsDiffer.get_component_diffs(from_dir, to_dir, for_git, dsp_from │
│ 168 │ │ if all(diff_status == ComponentsDiffer.DiffEnum.UNCHANGED for _, (diff_status, _ │
│ 169 │ │ │ raise UserWarning("Component is unchanged") │
│ 170 │ │ log.debug(f"Writing diff of '{component}' to '{diff_path}'") │
│ │
│ /Users/chrisata/miniforge3/envs/nextflow/lib/python3.13/site-packages/nf_core/components/compone │
│ nts_differ.py:100 in get_component_diffs │
│ │
│ 97 │ │ │ │
│ 98 │ │ │ elif temp_path.exists(): │
│ 99 │ │ │ │ with open(temp_path) as fh: │
│ ❱ 100 │ │ │ │ │ new_lines = fh.readlines() │
│ 101 │ │ │ │ # The file was created │
│ 102 │ │ │ │ # Show file against /dev/null │
│ 103 │ │ │ │ diff = difflib.unified_diff( │
│ in decode:325 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
It seems like binary/compressed files are trying to be read as regular text files from the subworkflow's unit tests, which is causing this error (see here: https://github.com/EBI-Metagenomics/nf-modules/tree/main/subworkflows/ebi-metagenomics/detect_rna/tests). I'm guessing it's either the test.fasta.gz or the Rfam covariance model files here (https://github.com/EBI-Metagenomics/nf-modules/tree/main/subworkflows/ebi-metagenomics/detect_rna/tests/rfam).
I've tried deleting these files locally in the pipeline and that doesn't solve the problem since it seems the patch command clones the repository into /Users/username/.config/nfcore/ebi-metagenomics/nf-modules which obviously still contains those files:
DEBUG Popen(['git', 'checkout', 'main'], cwd=/Users/chrisata/.config/nfcore/ebi-metagenomics/nf-modules, stdin=None, shell=False, cmd.py:1270
universal_newlines=False)
I've confirmed this by making a branch of the modules repo that doesn't have those files, installing the subworkflow from that branch, and finally trying to patch, and it generated a diff file with no issues (see here: https://github.com/EBI-Metagenomics/nf-modules/tree/test-remote-modules-diff).
Is this behaviour known, and if so are there plans to change it? It's quite limiting to not be able to have binary test files of any kind if you might want to run a patch one day.
Thanks so much!
Command used and terminal output
nf-core subworkflows --git-remote https://www.github.com/ebi-metagenomics/nf-modules patch detect_rna
System information
- nf-core/tools version: 3.3.2
- Python version: 3.13.8
- macOS Sonoma: 14.5
- Nextflow version: 25.04.8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the nf-core subworkflows --git-remote ... patch detect_rna command against the linked remote repository and inspect how the patch operation handles binary or compressed files. Done means the command completes successfully and produces the expected diff for a remote repository containing those files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- cli, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100