khanlab / khanlab/snakebids

`export_sourcedir` plugin

Open
#304 2 comments 0 reactions 0 assignees View on GitHub
enhancement plugin
Dominant language
Python
Stars
23
Forks
16
PR merge metrics
No merged PRs in 30d

Description

This proposal will supersede #180 .

The plugin will allow the copying or symlinking of directories in the app to the output dir. This will allow the snakemake app to access contained files via relative files, solving the provenance issues caused by reinstalling the app in a different location

```py
def export_sources(
paths: str | Path | list[str | Path],
*,
root: str | Path = ".",
timestamp: bool = True,
checksum: bool = False,
version: str | tuple[str, str, str] | packaging.version.Version | None = None,
symlink: bool = False,
):
"""Recursively copy given directories and files into the output dir.

Allows workflow to access files in the directories via relative paths. All exported
paths must be subdirectories of the root, which defaults to the snakemake_dir given
to SnakeBidsApp. Paths will be normalized relative to the root, then exported as-is.

If exported files are already in the output dir, the export will only be performed
on outdated files. By default, outdated files are determined by comparing the mtime
of the previously exported file to the file in the app. If a ``version``
specifier is used, checks for outdated files will only be made when the version
changes.

Parameters
==========
paths :
Paths of directories and files to export. Directories are exported recursively.
root :
Root of exported directories. All paths will be exported as relative to this
root.
timestamp :
Allow the default behaviour of updating exports whenever the exported copy is
older than the file to be exported. Defaults to ``True``.
checksum :
Find outdated exports by computing the file checksum. This is the most
robust solution to deal with snakemake's automatic provenance, but may be slow
for large files. If set, this takes precendence over ``timestamp``.
version :
A version string, tuple, or ``Version`` object to be recorded with the output
data. Outdated exports will be updated only when the version changes.
symlink :
If ``True``, symlinks will be made to exported files, rather than copying. This
method is faster when large files are being exported, but may not be compatible
with some distributed modes of snakemake.
"""
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the export_sources API proposal in this issue and compare it with the superseded #180. No implementation files or tests are named, so first locate the plugin entry point and existing output-export behavior. Done means supporting recursive copy or symlink export with the documented timestamp, checksum, version, and root semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.