Track pre-splitting a table in the job tracker
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper, I want to know when pre-splitting is happening, so that I know my job is still in progress, and I know the partitions will be split based on that job's data.
### Description / Background
Under epic:
- https://github.com/gchq/sleeper/issues/5078
In the following issue we added pre-splitting of a Sleeper table based on the data in a bulk import job:
- https://github.com/gchq/sleeper/issues/4881
When this is done, we'd like it to show up in the reports on ingest jobs.
### Technical Notes / Implementation Details
We can add this to IngestJobTracker. The new status update type can be a JobRunStatusUpdate. We can unit test this in InMemoryIngestJobTrackerTest. We'll need to support this in DynamoDBIngestJobTracker as well. We can test that in StoreIngestJobUpdatesIT.
This can be submitted to the ingest job tracker from BulkImportJobDriver or PartitionPreSplitter. We can unit test that in BulkImportJobDriverTest or PartitionPreSplitterTest.
Contributor guide
Research direction
Start with IngestJobTracker and JobRunStatusUpdate, then read InMemoryIngestJobTrackerTest and StoreIngestJobUpdatesIT to understand the existing update flow. Check BulkImportJobDriver and PartitionPreSplitter plus their named tests for where pre-splitting progress is submitted. Done means pre-splitting appears in ingest-job reports, with tracker and driver or pre-splitter coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100