Refactor and rethink the tabular ingest subsystem in v6.
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
[a placeholder for now, mostly]
I would like us to very seriously reconsider and possibly reimplement and/or refactor our system of ingesting and storing tabular data. It's a complicated setup, with a lot of technical debt. Some of the complexities are due to some legacy reasons that may not be very relevant anymore.
I'm creating this as an umbrella issue for keeping track of such ingest-related issues.
An example of an ingest-related implementation issue:
- #8524 (brand new)
Numerous issues where an option for skipping ingest is requested:
- #8525 (requested just now!)
- #6385
- #2199
- #7533
I would definitely like to finally have this implemented. The reasons it hasn't been implemented yet are kind of legacy too by now.
An example of an issue where specific ingestable format is discussed, and a need for a specific service offered on a tabular file is questioned:
- #7249
[I'll keep updating this list]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
This is an umbrella issue rather than a self-contained task. Start by reviewing issues #8524, #8525, #6385, #2199, #7533, and #7249 to map the ingest-related work; completion would require a defined scope and concrete implementation tasks for the tabular ingest redesign.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100