Convert standard ingest to use Spark
- Dominant language
- Java
- Stars
- 107
- Forks
- 29
- Avg merge
- 19h 46m
- Merged PRs (30d)
- 141
Description
### User Story
As a user of Sleeper, I want standard ingest and bulk import to behave the same, so that if my data works with standard ingest, it should work with bulk import as well.
### Description / Background
There are some differences between the way the Java Parquet reader reads data and the way Spark does. This can produce unexpected behaviour, e.g. handling of cases where a field is nullable in the input file but always has data.
We'd like to convert standard ingest to use Spark locally to ingest data. We could make this an option to begin with so we can compare performance.
### Technical Notes / Implementation Details
This could do with some analysis for which code would need to change and where we would inject the Spark implementation as an alternative.
Contributor guide
Research direction
Start by locating the standard ingest and bulk import entry points, then compare their Java Parquet reader and Spark-based paths. Identify where a Spark implementation could be injected as an option and how the two ingest modes can be compared. Done means standard ingest can use Spark locally and its behavior can be evaluated against bulk import, including the nullable-field case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100