getEvents: bloom filters load whole (8 MB) filter to read just a few KB

Summary

Under getEvents load, ~70% of CPU is the aggregated bloom filter read path, and ~46% is just decompressing 8 MB filter blobs. A query only uses 6 × (#terms) rows out of 8192, but we load and decompress the whole 8 MB matrix to reach them.

Fix: store each bloom row under its own key so a query reads only the rows it needs (~24 KB/window instead of 8 MB). Storage stays flat. The bloom logic doesn't change — only the on-disk layout.

Profile (getEvents saturation, random access)

Root cause

One filter per 8192-block window = a matrix of 8192 rows (bloom indices) × 8192 columns (blocks) = 8 MB. Stored as one key → one 8 MB value.

A query hashes each term to 6 row indices (bloom.Locations) and ANDs those rows to find candidate blocks. It needs ~24 rows. But since the matrix is one blob, reading 24 rows means loading and decompressing all 8192. ~99% is wasted.

The needed row indices come from hashing the query keys — known before any DB read.

Proposal

Change the layout only:

today:     (fromBlock, toBlock)     -> [8 MB matrix]
proposed:  (windowStart, rowIndex)  -> [1 row = 1 KB]   // skip empty rows

Query: compute the needed row indices (already done), fetch only those rows, run the same matching. A ~24-row query reads ~24–96 KB/window instead of 8 MB (~100–300× less).

Skip-empty rows: an unused bloom index isn't stored; a query miss means "no matches," zero read. Sparse filters get smaller than today.

Impact

• Removes most of the ~70% bloom path (decompress + reads + unmarshal).
• Storage flat (+~2–5% keys; same bitmap bytes; compression preserved) or smaller when sparse.
• Read cost scales with query selectivity, not filter size.

Trade-offs

• Many small gets instead of one read (mitigate: one iterator pass per window).
• More keys in the DB (~1M+ vs ~130).
• App-level filter cache needs redesign (it caches whole filters today).
• Read-path refactor + new "missing row = empty" handling.
• Reorg deletes many keys per window instead of one.
• Needs a migration to rewrite existing filters.

Running filter (current window) is in-memory and unaffected — only stored windows split.

Start at the getEvents bloom read path and trace the current stored-filter layout, cache handling, and reorg deletion behavior. Compare the existing whole-filter reads with the proposed per-window, per-row keys and identify the migration path for existing filters. Done means selective row reads preserve bloom matching, missing rows act as empty, and reorg and cache behavior remain correct.