typelevel / typelevel/frameless
Add support for TypedDeltaTable
Nobody has claimed this yet.
- Dominant language
- Scala
- Stars
- 895
- Forks
- 135
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 3
Description
Frameless introduced the TypedDataset on top of Apache Spark's Dataset API. The aim of this ticket is to create a TypedDeltaTable on top of delta.io's DeltaTable API. This would allow to have strongly typed merge actions, something that would look like the following:
val spark: SparkSession = ...
val updates: TypedDataset[U] = ...
val target: TypedDeltaTable[T] = TypedDataTable.unsafeForPath[T](spark, path)
val sq = target
.merge[U](updates, target('id) === updates('id))
.whenMatched(target('someSymbol) != updates('anotherSymbol))
.update(target('someSymbol) := updates('anotherSymbol))
.execute()
sq.awaitTermination()
This could be included in a new module, called frameless-delta.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing Frameless's existing TypedDataset API and delta.io's DeltaTable API, then determine how a new frameless-delta module would expose typed merge actions. Use the proposed TypedDeltaTable usage as the target shape; done means the module supports strongly typed merge operations over DeltaTable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100