bug: Can't read GCS files due to Decompressive Transcoding
- Dominant language
- Rust
- Stars
- 5.4k
- Forks
- 825
- Avg merge
- 1d 14m
- Merged PRs (30d)
- 127
Description
### Describe the bug
We have some files which are gzip encoded at rest in GCS. Thanks to [Decompressive transcoding](https://cloud.google.com/storage/docs/transcoding#decompressive_transcoding), when getting metadata about a file the compressed byte count will be returned, but when the file itself is retrieved it will be uncompressed and so the total byte count will be higher.
This interacts poorly with this PR: https://github.com/apache/opendal/pull/4690 where the two values mentioned above are compared and we keep hitting `reader got too much data`.
### Steps to Reproduce
I'm working on providing an actual repro but running into difficulties with permissions.
1. Have a file gzip encoded on GCS, with object metadata `Content-Encoding` set to `gzip`.
2. Now, build an `operator` of type `opendal::types::operator::operator::Operator` and call `operator.read()`
3. What currently happens is an `opendal::Error` as the `Result` that prints to:
```
Unexpected (permanent) at => reader got too much data
Context:
expect:
actual:
```
### Expected Behavior
An `Ok` `Result` with the contents of the file.
### Additional Context
_No response_
### Are you willing to submit a PR to fix this bug?
- [ ] Yes, I would like to submit a PR.
Contributor guide
Research direction
Start at the operator.read() entry point and review the behavior introduced by PR 4690. Reproduce the GCS case with a gzip-encoded object whose Content-Encoding is gzip, then verify that reading it returns an Ok Result containing the file contents rather than a reader-too-much-data error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, rust
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100