locationtech / locationtech/geomesa
FSDS - support sorted writes
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
FSDS - support sorted writes
If data files are sorted based on z-index, we can aggressively prune out files that don't match our filter on top of using the partition scheme. delta lake does this with spark repartitionByRange on a geohash column, look into how we could do this outside spark.
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2964
Key: GEOMESA-2964
Type: Improvement
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Thu, 10 Dec 2020 07:23:42 -0500
Updated: Thu, 17 Dec 2020 14:32:04 -0500
Comments:
557058:da7f6dfa-f0a1-468b-aec5-2cb61273feb5 (Thu, 17 Dec 2020 14:32:04 -0500):
may be useful for ideas: https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/planner/SortNode.java
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source file, test, or entry point is identified. Start by tracing the FSDS write path and comparing the issue's Spark repartitionByRange and Delta Lake references, then determine how sorted writes could work outside Spark. Done means implementing the agreed design and demonstrating that filters can prune non-matching files beyond the existing partition scheme.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100