nf-core / nf-core/website

docs: adding large datasets to s3://nf-core-awsmegatests

Open
#4,165 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation enhancement
Dominant language
MDX
Stars
98
Forks
298
Avg merge
14h 38m
Merged PRs (30d)
38

Description

Is your feature request related to a problem? Please describe.
We are working on releasing a pipeline which will require some large databases to be hosted on s3://nf-core-awsmegatests. There is a line mentioning "Request upload to nf-core AWS S3 test buckets in the [#request-core]" but its unclear what exactly. that entails.

Describe the solution you'd like
It would be great to have the same level of detail we have for uploading datasets to nf-core/test-datasets. Perhaps this could include

  • how to document the provenance of files we hope to upload
  • what tools are needed to do the transfer
  • whether any tests of the data need to be in place
  • limitations on file sizes
  • info on attribution for data
  • any information about versioning of databases

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by finding the existing nf-core/test-datasets upload documentation and the guidance linked from #request-core. Compare that process with uploads to s3://nf-core-awsmegatests, then document provenance, transfer tools, data tests, file-size limits, attribution, and database versioning so pipeline authors can follow the process without further clarification.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.