ArchiveBox / ArchiveBox/ArchiveBox
Enhancement: Add `WGET_EXTRA_ARGS`, `CURL_EXTRA_ARGS`, `SINGLEFILE_EXTRA_ARGS` to extend default args without overriding defaults
- Dominant language
- Python
- Stars
- 28.4k
- Forks
- 1.6k
- Avg merge
- 7h 54m
- Merged PRs (30d)
- 21
Description
These `WGET_ARGS`, `CURL_ARGS`, etc. options let the user shoot themselves in the foot, silently. I think the documentation or the variables themselves should be changed to be more ergonomic.
Did you need to add a header to wget?
```py
WGET_ARGS=['--header=Accept-Language: en-US,en']
```
Look at this configuration, it looks inoffensive. Checking [the documention](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#wget_args), nothing ticks you off you're using it wrong. **But no.**
After this, your 🆆 button won't take you to the main html you archived, and wget archives things slighlty differently, silently.
Why? By setting WGET_ARGS you overwrote [vital settings](https://github.com/ArchiveBox/ArchiveBox/blob/03eb7e58758d8dcb85ed781e713fc083f8292264/archivebox/config.py#L159), as it turns out they're also stored in WGET_ARGS. The documentation doesn't tell you about this. This happened to me, I just happily overwrote the variable. When I should've written something like this:
```py
WGET_ARGS=['--header=Accept-Language: en-US,en;q=0.5', '--no-verbose', '--adjust-extension', '--convert-links', '--force-directories', '--backup-converted', '--span-hosts', '--no-parent', '-e', 'robots=off']
```
Proposal:
1. The documentation for `*_ARGS` should have a good warning or display the default value, so that users can suspect they're overwriting something.
2. Or, create `EXTRA_WGET_ARGS`, `EXTRA_CURL_ARGS`, and so on. These won't overwrite the now-considered low-level `*_ARGS`. `EXTRA_*_ARGS` shall be the more user-facing option and more promoted in the documentation.
Contributor guide
Research direction
Start in archivebox/config.py at the WGET_ARGS defaults linked by the issue, then review the Configuration wiki section for WGET_ARGS. Define how the proposed EXTRA_*_ARGS variables relate to existing defaults for wget, curl, and SingleFile, and update the documentation so users know which settings preserve defaults; done means the new options extend rather than replace them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100