apache / apache/opendal

bug: Can't read GCS files due to Decompressive Transcoding

Open
#5,070 11 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
5.4k
Forks
825
Avg merge
1d 14m
Merged PRs (30d)
127

Description

### Describe the bug

We have some files which are gzip encoded at rest in GCS. Thanks to [Decompressive transcoding](https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding), when getting metadata about a file the compressed byte count will be returned, but when the file itself is retrieved it will be uncompressed and so the total byte count will be higher.

This interacts poorly with this PR: https://github.com/apache/opendal/pull/4690 where the two values mentioned above are compared and we keep hitting `reader got too much data`.

### Steps to Reproduce

I'm working on providing an actual repro but running into difficulties with permissions.

1. Have a file gzip encoded on GCS, with object metadata `Content-Encoding` set to `gzip`.
2. Now, build an `operator` of type `opendal::types::operator::operator::Operator` and call `operator.read()`
3. What currently happens is an `opendal::Error` as the `Result` that prints to:
```
Unexpected (permanent) at => reader got too much data

Context:

expect:

actual:
```

### Expected Behavior

An `Ok` `Result` with the contents of the file.

### Additional Context

_No response_

### Are you willing to submit a PR to fix this bug?

- [ ] Yes, I would like to submit a PR.

Contributor guide

Open the contributing guide

Research direction

Start at the operator.read() entry point and review the behavior introduced by PR 4690. Reproduce the GCS case with a gzip-encoded object whose Content-Encoding is gzip, then verify that reading it returns an Ok Result containing the file contents rather than a reader-too-much-data error.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, rust
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.