ozontech / ozontech/seq-db

Cache results of aggregation / histogram requests per sealed fraction

Open
#491 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
131
Forks
16
Avg merge
2d 4h
Merged PRs (30d)
11

Description

Dashboards re-issue the same aggregation/histogram queries every N seconds,
and seq-db recomputes them from scratch on every refresh — the most
expensive read path (docs/en/11-benchmarks.md) with the most repetitive
workload. Caching the results seems natural (cf. Elasticsearch's shard
request cache), and we'd like to know where the maintainers see it living:

  1. In seq-db (store level, per sealed fraction). Sealed fractions are
    immutable, so a result fully covering a fraction can be cached without
    TTL and stays exact; sliding windows reuse interior fractions. We built
    a prototype this way: an opt-in aggs layer in the existing cache
    subsystem, hit ≈ 2.7µs vs 14ms recompute on a 200k-doc fraction. Happy
    to submit a PR if this direction is acceptable.

  2. In seq-ui (Redis, TTL). Simpler, follows the existing
    logs_lifespan/events pattern, but results go stale within the TTL,
    a shifted time window is always a full miss, and other API consumers
    don't benefit.

Is per-fraction caching in seq-db something you'd accept, or do you consider
result caching out of scope for the engine and better kept in seq-ui?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/en/11-benchmarks.md and the existing cache subsystem to understand the aggregation and histogram read path and the stated performance gap. Compare that with the seq-ui logs_lifespan/events pattern; the issue is complete only after maintainers decide whether caching belongs in seq-db or seq-ui and define the accepted scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, go, redis
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.