internetarchive / internetarchive/warcprox
avoid clobbering existing warc
- Dominant language
- Python
- Stars
- 467
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
```
#first run
1.7M warc_cache/warcs/book.pythontips.com.warc.gz
#Second run, exact same code
516K warc_cache/warcs/book.pythontips.com.warc.gz
#Deleted dedupe but not warc file
1.7M warc_cache/warcs/book.pythontips.com.warc.gz
```
It looks like the dedupe file is used again, but the warc file is being created from scratch. That's definitely not was I would expect, is that how it's supposed to work? If you're recreating the warc file, shouldn't you be recreating the DB as well?
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named. Reproduce the documented first and second runs, then trace how the dedupe data and existing WARC path are handled; done means the behavior is defined and the WARC and dedupe state remain consistent when the cache is reused.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100