internetarchive / internetarchive/gowarc

No download limit - unused option MaxReadBeforeTruncate

Open
#102 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
55
Forks
12
Avg merge
10d 2h
Merged PRs (30d)
1

Description

gowarc has an option `MaxReadBeforeTruncate` with a default limit of 1GB. The problem is that this option isn’t used anywhere in gowarc and Zeno. Also, there was never such an option in `net/http`.

https://github.com/internetarchive/gowarc/blob/master/client.go#L139

This is a critical issue. A large target URL will OOM Zeno because when you use net/http to download a URL and you invoke `io.ReadAll()` it reads all in memory.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.