gchq / gchq/sleeper

Script to validate a file is suitable for ingest to a table

Open
#7,534 1 comment 0 reactions 0 assignees View on GitHub
enhancement scripts
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user, I want to be able to check if a file is suitable to ingest to a certain Sleeper table, so that I can tell if there are any problems with the file, or if the Sleeper table is not correctly configured.

### Description / Background

If a user creates a Sleeper table with the intention of adding particular data, they need to match up the Sleeper schema to the schema of the data in their input Parquet files.

Right now, in order to tell there's a mismatch, they need to submit their data to the Sleeper instance, and check the ingest job report to see if it fails. This involves a significant amount of waiting.

We'd like a script the user can use to check whether a data file matches the Sleeper table they want to add it to.

### Acceptance Criteria

**Given** I create a Sleeper table
**And** I have a Parquet file where one of the row key fields is nullable
**When** I run the script to check the file against the table
**Then** the script shows that that field is nullable in the file but not in the Sleeper table
**And** it explains any relevant behaviour in standard ingest vs bulk import

### Technical Notes / Implementation Details

This can go in the clients module as its own class, called directly from a script. It can use CommandArguments to parse the arguments. We can integration test it with a single entrypoint that takes the command line arguments, and reads an actual Parquet file and Sleeper table configuration.

We can use TablePropertiesStore to load the table configuration. We can use InMemoryTableProperties.getStore for tests.

The tests can use a real Parquet file in the local file system. It would be good if it could work with an S3 path as well, and we could test that against LocalStack as a separate test in its own class.

#### Different behaviour between bulk import and standard ingest

Note that the behaviour may be different between ingest and bulk import. Standard ingest uses our Parquet code to interact with the input files, but bulk import uses Spark. Spark might ignore the nullability of the input fields and only fail if there is actually a null value. We'd better verify this behaviour before implementing this script.

For the script, it would be good to explain to the user how this would apply differently for standard ingest and bulk import. It would be sufficient to only validate against the standard ingest behaviour initially.

Contributor guide

Open the contributing guide

Research direction

Start in the clients module and review CommandArguments, TablePropertiesStore, and InMemoryTableProperties.getStore to define the script entrypoint and test setup. Use an actual Parquet file and table configuration for integration coverage, with a separate LocalStack test for S3 paths if supported. Done means the script reports nullable row-key mismatches and explains the standard-ingest behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
66/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.