internetarchive / internetarchive/gowarc

Panic on too many open files error, should retry instead

Open
#43 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
55
Forks
12
Avg merge
10d 2h
Merged PRs (30d)
1

Description

Received the following error more than once:
```
panic: open jobs/warcs/TCPK-20240826191443122-00001-crawl918.us.archive.org.warc.gz.open: too many open files

goroutine 119 [running]:
github.com/CorentinB/warc.isFileSizeExceeded({0xc8816c7f40?, 0xc00041e3f0?}, 0x408f400000000000)
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/utils.go:240 +0xf5
github.com/CorentinB/warc.recordWriter(0xc00049e0f0, 0xc00014c070, 0xc0001b48c0, 0xc00041e3c0)
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/warc.go:137 +0x45a
created by github.com/CorentinB/warc.(*RotatorSettings).NewWARCRotator in goroutine 1
/var/www/go/pkg/mod/github.com/!corentin!b/warc@v0.8.43/warc.go:70 +0x8a
```

TODO: as stated in issue title, library should retry instead of panicking (according to @CorentinB)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the WARC rotation and file-opening path shown in warc.go and utils.go, especially the isFileSizeExceeded and recordWriter calls in the reported stack trace. Reproduce the too-many-open-files condition and verify that the library retries instead of panicking.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.