Partition S3 batch export files by event name
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
As a user of S3 batch exports, it can be easier to process individual files per event, but only partitioning by timestamp and table fields is currently supported.
The main challenge with this feature is that we do not know which event names to partition by before we query ClickHouse. So, we may need to delay the creation of an S3 upload until we start seeing events, and then maintain one S3 upload per event name. The question pending with this is how to allow recovery in the event of a worker crash now that we have a potentially very large number of simultaneous S3 uploads. Maybe Temporal heartbeating is not enough to support this feature and we will need to look into new ways of tracking progress.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing S3 batch export flow, including timestamp and table partitioning, the ClickHouse query, and Temporal worker recovery. Determine how event names can drive uploads and how progress can be recovered after a worker crash. Done means exports are partitioned by event name without losing recoverability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, clickhouse, python
- Domain
- backend, cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100