flashbots / flashbots/mempool-dumpster
Timestamp Sorting for Faster Data Access
- Dominant language
- Go
- Stars
- 259
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
## Current Problem
When we fetch mempool data (transactions), we don't sort transactions by timestamp. This makes it slow to find transactions within specific time ranges.
## Why This Matters
- Searching for transactions in a time range is much slower than it could be
- Each search has to scan through unsorted data
- This affects performance of time-based queries
## Simple Solution
Sort the data by timestamp when saving it. This will make time-based searches much faster. Also we could split it with smaller equal row_group_size like 5000
## Expected Benefits
- Much faster time-range searches
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the code that saves mempool transactions to Parquet and CSV, then trace how timestamps and row_group_size are handled. Determine how sorted data is queried for time ranges and validate that saving in timestamp order with the proposed row-group size improves those searches without breaking existing output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100