DTS integration for PATH parameter types
- Dominant language
- Java
- Stars
- 163
- Forks
- 65
- Avg merge
- 11h 46m
- Merged PRs (30d)
- 29
Description
Currently DTS integration handles only `INPUT` and `OUTPUT` parameters types.
If API determines that an `INPUT` param corresponds to the DTS location, dataset is transferred to the object storage container and then copied to the compute node for processing.
For some operations, we do not need the latter operation: only data upload from the DTS to the object storage container.
I propose to implement this using `PATH` parameters, which are not handled at all by the DTS/API:
1. If a DTS-related path is specified in the `PATH` parameter
2. Job initialization routine shall perform data transfer from the DTS-managed location to the transfer object storage
3. Update `PATH` parameter value (container environment) to the new object storage location (e.g. /projects/FASTQ/sample_1.fq -> s3://transfer-bucket/FASTQ/sample_1.fq)
4. Data copy from the object storage to the compute node FS shall not happen for this scenario
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.