epam / epam/cloud-pipeline

Add new fields to storage file object in Data Catalog

Open
#3,003 0 comments 0 reactions 2 assignees Claimed by @rodichenko View on GitHub
kind/enhancement
Dominant language
Java
Stars
163
Forks
65
Avg merge
11h 46m
Merged PRs (30d)
29

Description

**Background**
`Data Catalog` is a great tool to perform search on a big amount of indexed data stored in Cloud-Pipeline such as pipeline, datastorage or file inside a datastorage, etc.
To make this tool more powerful and user friendly it would be good to expand number of fields to index.
Currently `Data Catalog` already provides information on `path` for files stored in datastorages, however this path is relative to `datastorage path` and for some cases it can be not enough.

**Approach**
Let's in addition to `path` field also index fields `cloud_path` and `mount_path`:
- `cloud_path` - path to file with information of a datastorage: `s3://storage-name/path/to/file` `nfs://lustre.storage.address@tcp:/mount-id/path/to/file`
- `mount_path` - path to file with information of local mount point (how it will be mounted into a run): `/cloud-data/storage-name/path/to/file` `/custom-mount-point/path/to/file`

These fields will be sent on `/facet` and `/export` request, together with other fields like `id`, `name`, `path`, etc.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.