apache / apache/hudi-rs

Injecting `ObjectStore` for storage

Open
#330 3 comments 1 reaction 0 assignees View on GitHub
feature
Dominant language
Rust
Stars
279
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
40

Description

### Feature Description

Currently, a Hudi `Table` and the underlying `Storage` is constructed from the URL and storage options, and the storage options are kept in the struct. I found that `parse_url_opts` is used to determine the `ObjectStore` implementation for the URL, but this function only looks up built-in `ObjectStore` implementations from the `object_store` crate.

It would be great to support injecting `dyn ObjectStore` directly when constructing Hudi `Storage` and `Table`, since the user may have a custom `ObjectStore` implementation for custom URL schemes. Also I'm not sure if it's necessary to track storage options if the `ObjectStore` dependencies is injected from the outside.

Thank you!

### Additional context

I'm a maintainer of [Sail](https://github.com/lakehq/sail), a Rust library that offers a drop-in replacement for Apache Spark SQL and DataFrame APIs. We'd like to provide native Hudi support to our users. It would be great if we have the flexibility to integrate Hudi with the few custom `ObjectStore` implementations we have.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Table and Storage construction paths and the parse_url_opts entry point to understand how ObjectStore implementations and storage options are currently selected. Done means custom dyn ObjectStore implementations can be supplied for custom URL schemes, with the role of stored options clarified and the behavior covered by appropriate project tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.