apache / apache/arrow-rs-object-store

Object store AWS/S3 client and gRPC misconfiguration

Open
#35 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
322
Forks
212
Avg merge
5d 2h
Merged PRs (30d)
10

Description

**Describe the bug**
:warning: I understand that this is a very niche issue, but I thought I share this trap with others.

If you accidentally point an S3 `object_store` client to an gRPC endpoint, it will happily read empty objects for most paths (i.e. all paths that are not covered by the gRPC endpoint). This can become quite a debug nightmare.

**To Reproduce**
Set up a gRPC server, e.g. w/ `tonic`. The code example for the client sets this up to port `1234`.

Then configure an S3 Client to point at it. It's important that you
- [force the client to use HTTP/2](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_http2_only), otherwise it cannot connect to gRPC over HTTP/2 in the first place
- if your server doesn't run over TLS, you need to [allow HTTP](https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_allow_http).
- you may also want to [skip signature checks](https://docs.rs/object_store/latest/object_store/aws/struct.AmazonS3Builder.html#method.with_skip_signature)

```rust
let store = object_store::aws::AmazonS3Builder::new()
.with_bucket_name("dummy")
.with_client_options(
object_store::ClientOptions::new()
.with_allow_http(true)
.with_http2_only(),
)
.with_endpoint("http://localhost:1234")
.with_skip_signature(true)
.build()
.unwrap();
```

**Expected behavior**
I was naively expecting the client to error.

**Additional context**
gRPC for some bizarre reasons decides to not use the HTTP status code at all but instead a custom response header `grpc-status`. In our case, this is set to `12` for `UNIMPLEMENTED`, see
https://grpc.github.io/grpc/core/md_doc_statuscodes.html .

The response body for `UNIMPLEMENTED` is empty. The `content-length` response header is set to `0` (that's required by the `object_store` client).

:arrow_right: So I think what we could do as some kind of safeguard would be to check the `grpc-status` response header and bail out if it is set.

Contributor guide

Open the contributing guide

Research direction

The issue names no source file or test. Start by reproducing the S3 client configuration against a tonic gRPC endpoint, then trace how object_store handles the HTTP response and the grpc-status header. Done means an UNIMPLEMENTED gRPC response is reported as an error instead of being treated as an empty object.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, grpc, rust
Domain
api, backend, cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.