epam / epam/cloud-pipeline

Improve NFS search synchronization

Open
#2,155 0 comments 0 reactions 0 assignees View on GitHub
kind/enhancement sys/core sys/search
Dominant language
Java
Stars
163
Forks
65
Avg merge
11h 46m
Merged PRs (30d)
29

Description

**Background**
At the moment, NFS shares scanning is a time-consumable procedure, so synchronization of NFS storage is less frequent, compared to object storage.
As a result:
1. It is hard to control quotas on such type of storage
2. Its search relevance is low, which also affects the accuracy of billing reports.

It would be nice to implement a mechanism to reduce synchronization latency for NFS storage.

**Approach**
1. Let's implement a service, observing changes, affecting disk consumption of mounted shares.
To track all of the changes in a fileshares, that service should be active in all the runs and following core services:`cp-api-srv`, `cp-dav`.
It seems, that [inotify](https://man7.org/linux/man-pages/man7/inotify.7.html) API suits building such component.
2. Service gathers target FS events and persists them to configurable object storage.
3. Along with the regular full-synchronization of NFS, `elasticsearch-agent` performs more frequent resynchronization of files, affected by events collected.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the cp-api-srv and cp-dav services, then trace how elasticsearch-agent performs regular NFS synchronization. Evaluate the proposed inotify-based service, its configurable object-storage event persistence, and targeted resynchronization of affected files. Done means NFS changes are detected continuously and search, quota, and billing data become more current without replacing full synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, java
Domain
backend, distributed-systems, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.