apache / apache/arrow-rs-object-store

Better testing for different combinations of feature flags, different http clients and crypto libraries

Open
#759 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
As part of the 0.14.0 release, we are revamping how the feature flag system in object_store works, largely triggered by some changes in `reqwest`, the underlying http client library, way cryptography is handled

Specifically, as described in https://github.com/apache/arrow-rs-object-store/issues/744 / designed with ❤️ from @kevinjqliu , we will now support, via feature flags
1. Bring your own http cilent library (as well as supporting `reqwest` by default)
2. Bring you own crypto library (a well as supporting ring/aws-lc-rs)

This is a good thing as it allows users to customize the library exactly how they want, and minimize unecessary dependencies.

However, this design also results in a large number of feature flags; While adding tests for some combinations, it turns out it was fairly hard to test some of the combinations, as sometimes additional features are needed in other crates.

For example, to test "Run clippy with base features, reqwest and no bundled CryptoProvider" requires activating the `rustls-no-provider` feature of `reqwest

In this case, since `reqwest` is a direct dependency of object_store, we can do it

```shell
cargo clippy --no-default-features --features aws-base,azure-base,gcp-base,http-base,reqwest,reqwest/rustls-no-provider -- -D warnings
```

This makes it hard for me to understand what the test is actually testing and if it could be used this way by a downstream crate, especially as I mention in https://github.com/apache/arrow-rs-object-store/pull/707#issuecomment-4732460387

> I feel one difference is that actual customers will have their own Cargo.toml file where they can directly activate the features in the subcrates, so they won't be compiling just the object_store.

**Describe the solution you'd like**
I would like to
1. improve the testing of `object_store` with various feature flag combinations so it is clear what is tested and what is not
2. demonstrate / test that it is possible to use object_store with various flag combinations downstream to the desired effect

**Describe alternatives you've considered**

One way to test these features would be to create some targeted examples that are not part of the workspace and demonstrate (and verify they compile, etc) how to do the various usecases.

SO that would look like:
1. Remove the targeted clippy and `cargo tree` checks added in https://github.com/apache/arrow-rs-object-store/pull/707#issuecomment-4732460387
2. Add example crates like `example/custom_http_client/` and `example/custom_http_client_own_crypto` projects that are not part of the main workspace and that configures the feature flag in some way
3. Use the example to show: `cargo run` works, and that they don't bring in unwanted dependencies

**Additional context**
- Discusion: https://github.com/apache/arrow-rs-object-store/issues/744
- New feature flags, make reqwest optional: https://github.com/apache/arrow-rs-object-store/pull/724
- Reqwest update / feature flag disucssion: https://github.com/apache/arrow-rs-object-store/pull/707

Contributor guide

Open the contributing guide

Research direction

Review the targeted clippy and cargo tree checks from PR 707 alongside the feature-flag discussion in issue 744. Start by mapping the proposed example/custom_http_client/ and example/custom_http_client_own_crypto crates to downstream feature combinations; done means the examples compile and run while demonstrating the intended dependencies and coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
build-system, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.