Adding batch operation APIs for PinotFS
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently PinotFS doesn't support batch operation API extensions easily.
For some usage such as SegmentDeletionManager. once has to iterate over all files checking for deletion and then do the actual delete. This doesn't seem to be an issue with localFS, but for many of the cloud FS, there's more efficient batch APIs to reduce significantly on the remote request overheads.
Propose to
1. add `delete` / `copy` / `move` `(List segmentUris, ...)` API to its single URI variant.
2. make default implementation for them to fall back to looping each individually.
Contributor guide
Research direction
Start at the PinotFS abstraction and inspect the existing single-URI delete, copy, and move APIs and their implementations. Add list-based variants with default behavior that processes each URI individually, then verify that cloud filesystem implementations can provide efficient batch operations. Done means all three batch APIs are available without breaking existing single-URI usage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100