ozontech / ozontech/seq-db

Store .remote-info for remote shards to avoid extra S3 requests

Open
#435 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Go
Stars
131
Forks
16
Avg merge
2d 4h
Merged PRs (30d)
11

Description

Currently, for remote shards (uploaded to S3), only an empty file <shard-name>.remote exists locally. All .info data for such shards is retrieved either from .frac_cache (if cached) or directly from S3.

Problems
  1. Cannot determine shard format without an S3 request
    We cannot tell whether a shard is a single index or split into multiple files without making a request to S3.

  2. Losing the only local source of .info when removing .frac_cache
    If we remove .frac_cache, every store initialization will require fetching .info from S3 for each remote shard. This is unacceptable in terms of speed and cost.

Solution

For each remote shard, store a local file <shard-name>.remote-infoin exactly the same format as .info for regular shards.

Legacy shard format indicator:
info.BinaryDataVer < config.BinaryDataV3

Disk space estimate
  • Average .info size: 450–500 bytes
  • One shard every 5 minutes (after compaction)
    Yearly volume:
    500 bytes * (525,600 min / 5 min) ≈ 50 MB

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by locating remote-shard initialization and the existing .info, .remote, .frac_cache, and S3 handling paths; trace how regular-shard .info data is encoded. Done means each remote shard has a local .remote-info file in the same format, including the stated legacy-format indicator, so initialization can determine its format without an S3 request.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.