[Feature Request]: Add support for Delta Lake
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### What would you like to happen?
Using the new standalone driver of Databricks, it should be possible to support both reading and writing data to Delta Lake.
https://docs.delta.io/latest/delta-standalone.html
We can add Databricks server metadata to allow us to automatically configure the Hadoop Configuration objects with respect to connectivity. This might also be helpful for issue #2639 (DBFS).
The trick will be to re-use code and functionality of our Parquet Writer transform to avoid code duplication.
### Issue Priority
Priority: 3
### Issue Component
Component: Transforms
Contributor guide
Research direction
Start with the Delta Standalone documentation and inspect the existing Parquet Writer transform for reusable functionality. Define how reading and writing Delta Lake data and Databricks server metadata should configure Hadoop connectivity; done means both directions are supported without duplicating the Parquet Writer implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100