HttpURLConnection synchronously downloads complete response in memory
- Dominant language
- Java
- Stars
- 6k
- Forks
- 999
- Avg merge
- 19h 20m
- Merged PRs (30d)
- 14
Description
I am downloading large files and saving them. What caught me by surprise is that the `HttpURLConnection` fetches the whole response synchronously and in memory. This is problematic:
- You cannot download data from a URL while tracking the progress of the data retrieved thus far, as reading from the input stream blocks until the whole file was fetched
- All of this is handled in memory: when downloading large files memory usage rises significantly.
Is there perhaps another way to achieve this goal (download small chunks on the fly and outputting them) with the current J2Obc, or do we have to resort to native code?
Example code that won't behave as expected:
```
void download(String url, File dest, ProgressTracker tracker) throws IOException {
HttpURLConnection httpConnection = (HttpURLConnection) new URL(url).openConnection();
httpConnection.setRequestMethod("GET");
httpConnection.setDefaultUseCaches(false);
InputStream is = null;
OutputStream os = null;
try {
is = httpConnection.getInputStream();
os = new FileOutputStream(dest);
pipe(is, os, tracker); // reads `is` in blocks, while writing to `os`
} finally {
if (is != null) {
is.close();
}
if (os != null) {
os.close();
}
}
httpConnection.disconnect();
}
```
Contributor guide
Assessment
This issue has not been assessed yet.