epam / epam/cloud-pipeline

DTS integration for PATH parameter types

Open
#37 0 comments 0 reactions 1 assignee Claimed by @mzueva View on GitHub
kind/enhancement priority/low
Dominant language
Java
Stars
163
Forks
65
Avg merge
11h 46m
Merged PRs (30d)
29

Description

Currently DTS integration handles only `INPUT` and `OUTPUT` parameters types.

If API determines that an `INPUT` param corresponds to the DTS location, dataset is transferred to the object storage container and then copied to the compute node for processing.

For some operations, we do not need the latter operation: only data upload from the DTS to the object storage container.

I propose to implement this using `PATH` parameters, which are not handled at all by the DTS/API:
1. If a DTS-related path is specified in the `PATH` parameter
2. Job initialization routine shall perform data transfer from the DTS-managed location to the transfer object storage
3. Update `PATH` parameter value (container environment) to the new object storage location (e.g. /projects/FASTQ/sample_1.fq -> s3://transfer-bucket/FASTQ/sample_1.fq)
4. Data copy from the object storage to the compute node FS shall not happen for this scenario

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.