ResponseInputStream breaks GZIPInputStream on concatenated gzip streams (silent data truncation, v1 parity regression)
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 51
Description
**Related:** #4522 (closed for staleness — same underlying cause, never fixed)
## Description
When reading **concatenated gzip files** (multiple gzip members appended together, e.g., ALB access logs) from S3, wrapping the SDK v2 `ResponseInputStream` in `java.util.zip.GZIPInputStream` causes **silent data truncation**. The same code path worked correctly with SDK v1's `S3Object.getObjectContent()`.
This is a parity regression: migrating from v1 to v2 with no other code changes introduces non-deterministic data loss on concatenated gzip streams.
## Environment
- **AWS SDK v2 version:** 2.x (reproducible across recent versions)
- **Java version:** 11+
- **OS:** Linux (Amazon ECS / Fargate)
## Expected Behavior
Reading a concatenated gzip file from S3 through `GZIPInputStream` should return **all bytes from all gzip members**, identical to SDK v1 behavior.
A 318 MB concatenated gzip file (e.g., ALB logs with thousands of gzip members) should decompress fully regardless of buffer alignment.
## Actual Behavior
`GZIPInputStream` reads only a **partial, non-deterministic** amount of data — between 4 MB and 59 MB from a 318 MB file — and then returns `-1` (EOF), silently discarding the remaining gzip members.
The amount read varies between runs because the truncation depends on which internal buffer boundary a gzip member end happens to land on.
## Root Cause Analysis
`java.util.zip.GZIPInputStream` handles concatenated gzip streams by checking for additional members after each member ends. At member boundaries, it calls `in.available()` on the underlying `InflaterInputStream`, which in turn calls `in.available()` on the source stream. The logic in `GZIPInputStream` (see [OpenJDK source](https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/zip/GZIPInputStream.java)) interprets a `0` return from `available()` combined with a `-1` read as end-of-stream.
**SDK v1 (`S3ObjectInputStream`):** Wraps an Apache HTTP `InputStream` which implements `available()` in a way that `GZIPInputStream`'s concatenated-member detection works reliably. The underlying stream's buffering ensures bytes are typically "available" at member boundaries.
**SDK v2 (`ResponseInputStream`):** Uses a different HTTP client implementation (Netty-based or URL connection-based) where the `InputStream`'s `available()` semantics differ. At gzip member boundaries, when `GZIPInputStream` checks whether there's another member, the v2 stream can report `0` available (because the next chunk hasn't arrived from the network yet), causing `GZIPInputStream` to conclude the stream is exhausted.
This is timing-dependent: it occurs when a gzip member boundary aligns with an empty network buffer — which is common with large files over a network connection but rare in local/unit tests with `ByteArrayInputStream`.
## Reproduction
### Minimal setup
1. Create a concatenated gzip file (multiple gzip members appended):
```java
ByteArrayOutputStream baos = new ByteArrayOutputStream();
for (int i = 0; i < 1000; i++) {
GZIPOutputStream gzipOut = new GZIPOutputStream(baos);
gzipOut.write(("member " + i + "\n").getBytes());
gzipOut.finish();
gzipOut.flush();
}
// Upload baos.toByteArray() to S3
```
2. Read with SDK v2:
```java
S3Client s3 = S3Client.create();
ResponseInputStream ris = s3.getObject(
GetObjectRequest.builder().bucket("bucket").key("concatenated.gz").build()
);
BufferedInputStream bis = new BufferedInputStream(ris, 65536);
GZIPInputStream gzis = new GZIPInputStream(bis, 65536);
byte[] buf = new byte[8192];
long total = 0;
int n;
while ((n = gzis.read(buf)) != -1) {
total += n;
}
// 'total' will be LESS than the full decompressed size on large files
// The exact amount varies per run (non-deterministic)
```
**Note:** This is difficult to reproduce with small files or local streams because it requires the network read to return 0 bytes available at a gzip member boundary. Use a file with many (hundreds+) gzip members totaling >100 MB to reliably trigger the issue.
### Production evidence
Migrating an S3 reader from SDK v1 to v2 with no other changes:
| SDK | File size (compressed) | Bytes decompressed | Result |
|-----|----------------------|-------------------|--------|
| v1 (`AmazonS3.getObject()`) | 318 MB | 318,926,634 (full) | Correct |
| v2 (`S3Client.getObject()`) | 318 MB | 4–59 MB (varies) | **Truncated** |
## Workaround
Replace `java.util.zip.GZIPInputStream` with Apache Commons Compress `GzipCompressorInputStream(stream, true)`, which handles concatenated members without relying on `available()` semantics:
```java
import org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;
ResponseInputStream ris = s3.getObject(request);
BufferedInputStream bis = new BufferedInputStream(ris, 65536);
GzipCompressorInputStream gzis = new GzipCompressorInputStream(bis, true);
// reads all concatenated members correctly
```
## Suggested Fix
The SDK v2 `ResponseInputStream` (or its underlying `AbortableInputStream`) should ensure that `available()` does not return `0` when there are unread bytes remaining in the stream, or at minimum should match the `available()` contract that SDK v1's stream provided.
Possible approaches:
1. **Override `available()`** in `ResponseInputStream` to return `1` when the stream is not exhausted (consistent with the `InputStream` contract which states `available()` returning `0` does not mean EOF, but `GZIPInputStream` uses it as a heuristic).
2. **Wrap the response stream in a `BufferedInputStream`** internally so that `available()` reflects buffered bytes, matching v1 behavior.
3. **Document the incompatibility** in the v1→v2 migration guide with explicit guidance to use `GzipCompressorInputStream` for concatenated gzip streams.
## Impact
Any application that:
- Reads concatenated gzip files from S3 (ALB logs, CloudFront logs, custom log pipelines)
- Uses `java.util.zip.GZIPInputStream`
- Migrated from SDK v1 to v2
...is silently losing data. The non-deterministic nature makes it difficult to detect — the stream appears to complete successfully, just with fewer bytes.
## References
- [OpenJDK GZIPInputStream concatenated member handling](https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/util/zip/GZIPInputStream.java)
- [RFC 1952 §2.2](https://datatracker.ietf.org/doc/html/rfc1952#section-2.2): "A gzip file consists of a series of 'members' (compressed data sets). [...] The members simply appear one after another in the file"
- [AWS SDK v1 S3ObjectInputStream](https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-s3/src/main/java/com/amazonaws/services/s3/model/S3ObjectInputStream.java)
- [AWS SDK v2 ResponseInputStream](https://github.com/aws/aws-sdk-java-v2/blob/master/core/sdk-core/src/main/java/software/amazon/awssdk/core/ResponseInputStream.java)
Contributor guide
Research direction
Start with core/sdk-core/src/main/java/software/amazon/awssdk/core/ResponseInputStream.java and the linked OpenJDK GZIPInputStream behavior; run the large concatenated-member S3 reproduction with Java 11+ and a BufferedInputStream. Compare available() and decompressed byte counts at gzip member boundaries. Done means the full concatenated stream is read consistently, with regression coverage for the reported truncation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100