Caching of DataFrame between processing phases
Offen
cloud
Conformance
feature
priority: undecided
Standardization
under discussion
- Vorherrschende Sprache
- Scala
- Sterne
- 33
- Forks
- 16
- PR-Merge-Kennzahlen
- Keine gemergten PRs in 30 T.
Beschreibung
## Background
Right now, based on #1538, the DataFrame is re-read between each phase. This is potentially ineffective and results it extra IO (possible extra cost in cloud)
## Feature
Investigate and eventually implement some Spark caching strategy resistant to node failure.
## Proposed Solution
This might be a source reading for possible solution
https://sparkbyexamples.com/spark/spark-dataframe-cache-and-persist-explained/
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.