aws / aws/aws-sdk-java-v2

S3: getObject, ResponseInputStream close not working as expected.

Open
#2,117 6 comments 3 reactions 0 assignees View on GitHub
bug p3
Dominant language
Java
Stars
2.6k
Forks
1k
Avg merge
2d 9h
Merged PRs (30d)
51

Description

## Describe the bug
When you invoke close on ResponseInputStream from the getObject before you reach the end of the data. The stream will not immediately(/within reasonable time) close on big files.

## Expected Behavior
When invoking close on an ResponseInputStream you would expect a close within at least a few seconds.

## Current Behavior
It does not close until it did a complete read of the stream internally. (this seems related to connection recycling internally).

## Steps to Reproduce
- Upload a big file to S3 atleast a 1-2 gb to observe the issue on fast internet connection
```
var r = client.getObject(c -> {
c.bucket(bucket).key(key).versionId(versionId);
});
r.read(new byte[1024]); // just to make sure we actually opened the resource and did something.
r.close(); // now you have to wait until the entire file is read by the close internally.
```

## Possible Solution
- The internal handler closes the stream immediately, possibly by simply closing the underlying connection.
- It could make a decision based on the remaining bytes to read to make the impact less significant on big files (e.g. if remaining > 1 MB close connection, otherwise consume).

## Context
I was trying to create a test case for a wrapper i created for handling network related issues (e.g. connection reset). Where i would simply track the position in the stream and use a range request when the connection fails, to continue the stream transparently.
When testing it by creating a wrapper around the ResponseInputStream that throws randomly a IO exception i noticed the extreme delay when closing the ResponseInputStream.

## Workaround
```
var r = client.getObject(c -> {
c.bucket(bucket).key(key).versionId(versionId);
});
r.read(new byte[1024]); // just to make sure we actually opened the resource and did something.
r.abort(); // <<< Workaround, but this is not part of a InputStream contract
r.close(); // now the connection closes almost immediately.
```
## Your Environment

* AWS Java SDK version used: 2.15.14
* JDK version used: 11
* Operating System and version: windows 10

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.