internetarchive / internetarchive/gowarc
Panic on too many open files error, should retry instead
- Dominant language
- Go
- Stars
- 55
- Forks
- 12
- Avg merge
- 10d 2h
- Merged PRs (30d)
- 1
Description
Received the following error more than once:
```
panic: open jobs/warcs/TCPK-20240826191443122-00001-crawl918.us.archive.org.warc.gz.open: too many open files
goroutine 119 [running]:
github.com/CorentinB/warc.isFileSizeExceeded({0xc8816c7f40?, 0xc00041e3f0?}, 0x408f400000000000)
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/utils.go:240 +0xf5
github.com/CorentinB/warc.recordWriter(0xc00049e0f0, 0xc00014c070, 0xc0001b48c0, 0xc00041e3c0)
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/warc.go:137 +0x45a
created by github.com/CorentinB/warc.(*RotatorSettings).NewWARCRotator in goroutine 1
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/warc.go:70 +0x8a
```
TODO: as stated in issue title, library should retry instead of panicking (according to @CorentinB)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the WARC rotation and file-opening path shown in warc.go and utils.go, especially the isFileSizeExceeded and recordWriter calls in the reported stack trace. Reproduce the too-many-open-files condition and verify that the library retries instead of panicking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100