Store .remote-info for remote shards to avoid extra S3 requests
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 131
- Forks
- 16
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 11
Description
Currently, for remote shards (uploaded to S3), only an empty file <shard-name>.remote exists locally. All .info data for such shards is retrieved either from .frac_cache (if cached) or directly from S3.
Problems
-
Cannot determine shard format without an S3 request
We cannot tell whether a shard is a single index or split into multiple files without making a request to S3. -
Losing the only local source of
.infowhen removing.frac_cache
If we remove.frac_cache, every store initialization will require fetching.infofrom S3 for each remote shard. This is unacceptable in terms of speed and cost.
Solution
For each remote shard, store a local file <shard-name>.remote-info – in exactly the same format as .info for regular shards.
Legacy shard format indicator:
info.BinaryDataVer < config.BinaryDataV3
Disk space estimate
- Average
.infosize: 450–500 bytes - One shard every 5 minutes (after compaction)
Yearly volume:
500 bytes * (525,600 min / 5 min) ≈ 50 MB
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating remote-shard initialization and the existing .info, .remote, .frac_cache, and S3 handling paths; trace how regular-shard .info data is encoded. Done means each remote shard has a local .remote-info file in the same format, including the stated legacy-format indicator, so initialization can determine its format without an S3 request.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100