apache / apache/arrow-rs-object-store
GCS: `get()` fails with `MissingContentLength` on chunked responses (`Content-Encoding: gzip`)
- Dominant language
- Rust
- Stars
- 322
- Forks
- 212
- Avg merge
- 5d 2h
- Merged PRs (30d)
- 10
Description
**Describe the bug**
`ObjectStore::get` on the GCS backend fails with `Error::Generic { store: "GCS", source: Header { source: MissingContentLength } }` for any object stored with `Content-Encoding: gzip`. GCS serves these via `Transfer-Encoding: chunked` with **no `Content-Length`**, and object_store treats a missing `Content-Length` as fatal.
It manifests in two ways, and crucially **a client cannot fully avoid it**:
- **Default reads (no `Accept-Encoding`):** GCS applies *decompressive transcoding* — it decompresses the object server-side and streams the result chunked, with no `Content-Length`. Every gzip object fails, at any size.
- **Even with `Accept-Encoding: gzip`** (which returns the raw stored bytes): objects whose **stored size exceeds ~8 MiB** are *still* served chunked with no `Content-Length` (empirically the cutover is between 9 and 10 MB), so they fail too. Only *small* objects read with `Accept-Encoding: gzip` succeed.
So whether you receive the transcoded (uncompressed) bytes or the raw (compressed) bytes, the response is chunked with no `Content-Length` — and object_store rejects it before returning any data. `head()` and range reads on gzip objects fail as well.
These are valid HTTP/1.1 responses: a chunked body is self-delimiting, and [RFC 9112 §6.2](https://www.rfc-editor.org/rfc/rfc9112#section-6.2) states a sender **MUST NOT** send `Content-Length` together with `Transfer-Encoding`;
[RFC 9112 §6.3](https://www.rfc-editor.org/rfc/rfc9112#section-6.3) says the chunked framing determines the body length and overrides any `Content-Length`. object_store is therefore requiring a header the spec forbids on exactly these responses.
**To Reproduce**
Observe the offending response with no credentials against Google's public demo data (the `?cb=` cache-buster forces origin past the public edge cache):
```console
$ curl -sD - -o /dev/null \
"https://storage.googleapis.com/neuroglancer-public-data/kasthuri2011/ground_truth/6_6_30/5376-5440_6656-6720_896-960?cb=$RANDOM" \
| grep -iE "HTTP/|content-length|transfer-encoding|x-goog-stored-content-encoding"
HTTP/2 200
transfer-encoding: chunked
x-goog-stored-content-encoding: gzip
# (no content-length)
```
object_store then fails on that same public object (authenticated read — a service account or on GCP; verified with object_store 0.13.1 and main `de0029a`):
```rust
use object_store::gcp::GoogleCloudStorageBuilder;
use object_store::{path::Path, ObjectStore, ObjectStoreExt};
let store = GoogleCloudStorageBuilder::new()
.with_bucket_name("neuroglancer-public-data")
.with_service_account_path(sa) // or workload identity on GCP
.build()?;
store
.get(&Path::from(
"kasthuri2011/ground_truth/6_6_30/5376-5440_6656-6720_896-960",
))
.await?; // Err: Generic { store: "GCS", source: Header { source: MissingContentLength } }
```
For a self-contained object of any size in your own bucket (a >8 MiB one fails even with `Accept-Encoding: gzip`):
```console
head -c 20000000 /dev/urandom | gzip | gsutil -h "Content-Encoding:gzip" cp - gs://YOUR_BUCKET/big.gz
```
> **Note:** the bundled `fake-gcs-server` does **not** emulate decompressive transcoding (it always
> returns a `Content-Length`), so it cannot reproduce this. The crate's own `MockServer` can — push a
> response with a chunked body and no `Content-Length` header.
**Expected behavior**
A full GET of a valid, self-delimiting (chunked / HTTP-2) response should succeed by reading the body to completion rather than requiring a `Content-Length` header that the HTTP spec forbids on chunked responses.
**Additional context**
- Affects gzip only; `br`/`zstd`/uncompressed objects are served identity (with `Content-Length`) and read fine.
- Root cause: `header_meta` requires `CONTENT_LENGTH` unconditionally ([`header.rs#L144`](https://github.com/apache/arrow-rs-object-store/blob/de0029aa91f7727015fab37e623fdbe11672914b/src/client/header.rs#L144), in [`header_meta` L114](https://github.com/apache/arrow-rs-object-store/blob/de0029aa91f7727015fab37e623fdbe11672914b/src/client/header.rs#L114)), and the GET path derives `ObjectMeta.size`/`range` from it before streaming
([`get.rs#L314`](https://github.com/apache/arrow-rs-object-store/blob/de0029aa91f7727015fab37e623fdbe11672914b/src/client/get.rs#L314) → [`#L333`](https://github.com/apache/arrow-rs-object-store/blob/de0029aa91f7727015fab37e623fdbe11672914b/src/client/get.rs#L333)).
- Wire evidence (>8 MiB gzip object, authenticated, `Accept-Encoding: gzip`):
`Transfer-Encoding: chunked`, no `Content-Length`, `x-goog-stored-content-encoding: gzip`,
`x-goog-stored-content-length: `.
Prior art — every other major GCS client reads to EOF (commit-pinned)
| client | behavior | source |
|---|---|---|
| google-cloud-storage (Python) | streams via `response.iter_content()`; `x-goog-stored-content-length` only a retry heuristic | [download.py#L145](https://github.com/googleapis/python-storage/blob/ab4997ce0f7b85947e84b226bd0edf6d714a946a/google/cloud/storage/_media/requests/download.py#L145), [#L163](https://github.com/googleapis/python-storage/blob/ab4997ce0f7b85947e84b226bd0edf6d714a946a/google/cloud/storage/_media/requests/download.py#L163) |
| cloud.google.com/go/storage (Go) | `Reader.Remain()` returns `-1` for chunked / `Decompressed`; CRC skipped on transcoding | [reader.go#L436](https://github.com/googleapis/google-cloud-go/blob/a25e93d25635b8fd42985edbe0290ba9a8cf2169/storage/reader.go#L436), [#L74](https://github.com/googleapis/google-cloud-go/blob/a25e93d25635b8fd42985edbe0290ba9a8cf2169/storage/reader.go#L74), [http_client.go#L1455](https://github.com/googleapis/google-cloud-go/blob/a25e93d25635b8fd42985edbe0290ba9a8cf2169/storage/http_client.go#L1455) |
| google-cloud-cpp (Apache Arrow C++) | reads to EOF via `HasUnreadData()`; **falls back to `x-goog-stored-content-length`** for size | [object_read_source.cc#L114](https://github.com/googleapis/google-cloud-cpp/blob/149ca440cc492a66e612e2e1f1fb385136530110/google/cloud/storage/internal/rest/object_read_source.cc#L114), [#L55](https://github.com/googleapis/google-cloud-cpp/blob/149ca440cc492a66e612e2e1f1fb385136530110/google/cloud/storage/internal/rest/object_read_source.cc#L55) |
| TensorStore (C++ GCS kvstore) | accumulates body via libcurl callback to EOF; enables Accept-Encoding | [gcs_key_value_store.cc#L578](https://github.com/google/tensorstore/blob/613280f459520c7dddc9aa11a41412a0c2a6b913/tensorstore/kvstore/gcs_http/gcs_key_value_store.cc#L578) |
| gcsfs (fsspec) | reads the aiohttp stream to EOF | [core.py#L118](https://github.com/fsspec/gcsfs/blob/f707e61fa75dcb4dc6b7bad0bc2321d425336a3a/gcsfs/core.py#L118) |
| rclone (Go) | GCS backend returns `res.Body`, reads to EOF | [googlecloudstorage.go#L1385](https://github.com/rclone/rclone/blob/59c86b01bb39624650badd39f3acfd20be2b743b/backend/googlecloudstorage/googlecloudstorage.go#L1385), [issue #2658](https://github.com/rclone/rclone/issues/2658) |
| smart_open (Python) | same bug class, fixed by delegating to google-cloud-storage | [issue #422](https://github.com/piskvorky/smart_open/issues/422) |
**AI disclaimer:**
* 🤷♂️ ran into the bug
* 🤖 did initial investigation
* 🤷♂️ verified investigation results (checked RFC, response headers, and `google-cloud-storage` behavior)
* 🤖 helped writing the report
* 🤷♂️ proofread and made funny AI disclaimer list
Contributor guide
Research direction
Start in src/client/header.rs at header_meta and follow the GET path in src/client/get.rs where ObjectMeta.size and range are derived. Use the crate's MockServer to exercise a chunked response without Content-Length, then inspect the existing tests around these client paths. Done means valid chunked or HTTP/2 GET responses stream to completion without MissingContentLength.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rust
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100