locationtech / locationtech/geomesa
FSDS Spark - support overwrite by partition persistence
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 1.5k
- Forks
- 446
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 31
Description
FSDS Spark - support overwrite by partition persistence
From geoHeil geoHeil on gitter:
spark.conf.set("spark.sql.sources.partitionOverwriteMode","dynamic")
data.write.mode("overwrite").insertInto("partitioned_table")
The goal is to allow loading the data through idempotent jobs (self healing) which greatly simplifies workflows.
In the FSDS, we could overwrite per partition in the partition scheme. Note that we don't have to physically delete the existing files on disk, we should be able to just mark them deleted in the metadata.
Original JIRA Issue: https://geomesa.atlassian.net/browse/GEOMESA-2531
Key: GEOMESA-2531
Type: Story
Priority: Major
Status: To Do
Resolution: Unresolved
Reporter: Emilio Lahr-Vivaz
Created: Thu, 24 Jan 2019 08:51:48 -0500
Updated: Fri, 25 Jan 2019 09:14:35 -0500
JIRA Labels: fsds
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the FSDS partition scheme and the Spark overwrite behavior described in the linked Stack Overflow question. Determine how partition-scoped overwrites should interact with persisted metadata rather than physical file deletion. Done means idempotent loads can replace only the affected partitions and mark prior data deleted in metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100