apache / apache/arrow-rs-object-store
Object store AWS/S3 client and gRPC misconfiguration
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
:warning: I understand that this is a very niche issue, but I thought I share this trap with others.
If you accidentally point an S3 `object_store` client to an gRPC endpoint, it will happily read empty objects for most paths (i.e. all paths that are not covered by the gRPC endpoint). This can become quite a debug nightmare.
**To Reproduce**
Set up a gRPC server, e.g. w/ `tonic`. The code example for the client sets this up to port `1234`.
Then configure an S3 Client to point at it. It's important that you
- [force the client to use HTTP/2](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_http2_only), otherwise it cannot connect to gRPC over HTTP/2 in the first place
- if your server doesn't run over TLS, you need to [allow HTTP](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_allow_http).
- you may also want to [skip signature checks](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_skip_signature)
```rust
let store = object_store::aws::AmazonS3Builder::new()
.with_bucket_name("dummy")
.with_client_options(
object_store::ClientOptions::new()
.with_allow_http(true)
.with_http2_only(),
)
.with_endpoint("http://localhost:1234")
.with_skip_signature(true)
.build()
.unwrap();
```
**Expected behavior**
I was naively expecting the client to error.
**Additional context**
gRPC for some bizarre reasons decides to not use the HTTP status code at all but instead a custom response header `grpc-status`. In our case, this is set to `12` for `UNIMPLEMENTED`, see
https://grpc.github.io/grpc/core/md_doc_statuscodes.html .
The response body for `UNIMPLEMENTED` is empty. The `content-length` response header is set to `0` (that's required by the `object_store` client).
:arrow_right: So I think what we could do as some kind of safeguard would be to check the `grpc-status` response header and bail out if it is set.
Contributor guide
Research direction
The issue names no source file or test. Start by reproducing the S3 client configuration against a tonic gRPC endpoint, then trace how object_store handles the HTTP response and the grpc-status header. Done means an UNIMPLEMENTED gRPC response is reported as an error instead of being treated as an empty object.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, grpc, rust
- Domain
- api, backend, cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100