gchq / gchq/sleeper

Track pre-splitting a table in the job tracker

Open
#6,607 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want to know when pre-splitting is happening, so that I know my job is still in progress, and I know the partitions will be split based on that job's data.

### Description / Background

Under epic:
- https://github.com/gchq/sleeper/issues/5078

In the following issue we added pre-splitting of a Sleeper table based on the data in a bulk import job:
- https://github.com/gchq/sleeper/issues/4881

When this is done, we'd like it to show up in the reports on ingest jobs.

### Technical Notes / Implementation Details

We can add this to IngestJobTracker. The new status update type can be a JobRunStatusUpdate. We can unit test this in InMemoryIngestJobTrackerTest. We'll need to support this in DynamoDBIngestJobTracker as well. We can test that in StoreIngestJobUpdatesIT.

This can be submitted to the ingest job tracker from BulkImportJobDriver or PartitionPreSplitter. We can unit test that in BulkImportJobDriverTest or PartitionPreSplitterTest.

Contributor guide

Open the contributing guide

Research direction

Start with IngestJobTracker and JobRunStatusUpdate, then read InMemoryIngestJobTrackerTest and StoreIngestJobUpdatesIT to understand the existing update flow. Check BulkImportJobDriver and PartitionPreSplitter plus their named tests for where pre-splitting progress is submitted. Done means pre-splitting appears in ingest-job reports, with tracker and driver or pre-splitter coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.