PostgreSQL writer
- Dominant language
- Scala
- Stars
- 81
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
Current implemented for streaming sources as a `foreachWriter`, but would be more general if it were actually a sink.
Sources of inspiration:
* https://github.com/phatak-dev/spark2.0-examples/blob/master/src/main/scala/com/madhukaraphatak/examples/sparktwo/datasourcev2/MysqlWithTransaction.scala
* https://www.slideshare.net/datamantra/understanding-transactional-writes-in-datasource-v2-109643509
* http://blog.madhukaraphatak.com/spark-datasource-v2-part-6/
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the current PostgreSQL implementation for streaming sources and its foreachWriter integration. Read the linked Spark DataSource V2 transaction examples, then define completion as replacing or generalizing the writer so it functions as a sink rather than only through foreachWriter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, scala, spark
- Domain
- data-engineering, databases
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100