gchq / gchq/sleeper

Convert standard ingest to use Spark

Open
#7,537 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
107
Forks
29
Avg merge
19h 46m
Merged PRs (30d)
141

Description

### User Story

As a user of Sleeper, I want standard ingest and bulk import to behave the same, so that if my data works with standard ingest, it should work with bulk import as well.

### Description / Background

There are some differences between the way the Java Parquet reader reads data and the way Spark does. This can produce unexpected behaviour, e.g. handling of cases where a field is nullable in the input file but always has data.

We'd like to convert standard ingest to use Spark locally to ingest data. We could make this an option to begin with so we can compare performance.

### Technical Notes / Implementation Details

This could do with some analysis for which code would need to change and where we would inject the Spark implementation as an alternative.

Contributor guide

Open the contributing guide

Research direction

Start by locating the standard ingest and bulk import entry points, then compare their Java Parquet reader and Spark-based paths. Identify where a Spark implementation could be injected as an option and how the two ingest modes can be compared. Done means standard ingest can use Spark locally and its behavior can be evaluated against bulk import, including the nullable-field case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.