DuckDB compatibility with HPC resources
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 148
- Forks
- 55
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 18
Description
Not all HPCs (ex. ZIB) are set up to work with DuckDB download settings. Proposed fix TBD
Current temp fixes are:
- Manual download and
snakemake --touchto force use of manually downloaded files - Adding proxy settings. Specifically for ZIB cluster:
duckdb.query("SET http_proxy='squid.zib.de:3128';")```
http_proxy could be parameterized from the environment variable `http_proxy` and this is probably the best long term solution to be implemented in a PR
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the DuckDB download setup and the workflow entry point that currently requires manual downloads or snakemake --touch; no file or test is named in the issue. Read how proxy settings are passed to DuckDB, then verify that the relevant HPC download path can use the http_proxy environment variable without manual intervention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100