Injecting `ObjectStore` for storage
- Dominant language
- Rust
- Stars
- 279
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 40
Description
### Feature Description
Currently, a Hudi `Table` and the underlying `Storage` is constructed from the URL and storage options, and the storage options are kept in the struct. I found that `parse_url_opts` is used to determine the `ObjectStore` implementation for the URL, but this function only looks up built-in `ObjectStore` implementations from the `object_store` crate.
It would be great to support injecting `dyn ObjectStore` directly when constructing Hudi `Storage` and `Table`, since the user may have a custom `ObjectStore` implementation for custom URL schemes. Also I'm not sure if it's necessary to track storage options if the `ObjectStore` dependencies is injected from the outside.
Thank you!
### Additional context
I'm a maintainer of [Sail](https://github.com/lakehq/sail), a Rust library that offers a drop-in replacement for Apache Spark SQL and DataFrame APIs. We'd like to provide native Hudi support to our users. It would be great if we have the flexibility to integrate Hudi with the few custom `ObjectStore` implementations we have.
Contributor guide
Research direction
Start by tracing the Table and Storage construction paths and the parse_url_opts entry point to understand how ObjectStore implementations and storage options are currently selected. Done means custom dyn ObjectStore implementations can be supplied for custom URL schemes, with the role of stored options clarified and the behavior covered by appropriate project tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100