Improve NFS search synchronization
- Dominant language
- Java
- Stars
- 163
- Forks
- 65
- Avg merge
- 11h 46m
- Merged PRs (30d)
- 29
Description
**Background**
At the moment, NFS shares scanning is a time-consumable procedure, so synchronization of NFS storage is less frequent, compared to object storage.
As a result:
1. It is hard to control quotas on such type of storage
2. Its search relevance is low, which also affects the accuracy of billing reports.
It would be nice to implement a mechanism to reduce synchronization latency for NFS storage.
**Approach**
1. Let's implement a service, observing changes, affecting disk consumption of mounted shares.
To track all of the changes in a fileshares, that service should be active in all the runs and following core services:`cp-api-srv`, `cp-dav`.
It seems, that [inotify](https://man7.org/linux/man-pages/man7/inotify.7.html) API suits building such component.
2. Service gathers target FS events and persists them to configurable object storage.
3. Along with the regular full-synchronization of NFS, `elasticsearch-agent` performs more frequent resynchronization of files, affected by events collected.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the cp-api-srv and cp-dav services, then trace how elasticsearch-agent performs regular NFS synchronization. Evaluate the proposed inotify-based service, its configurable object-storage event persistence, and targeted resynchronization of affected files. Done means NFS changes are detected continuously and search, quota, and billing data become more current without replacing full synchronization.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch, java
- Domain
- backend, distributed-systems, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100