azavea / azavea/osmesa

PostgreSQL writer

Open
#93 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
81
Forks
26
PR merge metrics
No merged PRs in 30d

Description

Current implemented for streaming sources as a `foreachWriter`, but would be more general if it were actually a sink.

Sources of inspiration:

* https://github.com/phatak-dev/spark2.0-examples/blob/master/src/main/scala/com/madhukaraphatak/examples/sparktwo/datasourcev2/MysqlWithTransaction.scala
* https://www.slideshare.net/datamantra/understanding-transactional-writes-in-datasource-v2-109643509
* http://blog.madhukaraphatak.com/spark-datasource-v2-part-6/

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the current PostgreSQL implementation for streaming sources and its foreachWriter integration. Read the linked Spark DataSource V2 transaction examples, then define completion as replacing or generalizing the writer so it functions as a sink rather than only through foreachWriter.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, scala, spark
Domain
data-engineering, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.