ArchiveBox / ArchiveBox/ArchiveBox

Enhancement: Add `WGET_EXTRA_ARGS`, `CURL_EXTRA_ARGS`, `SINGLEFILE_EXTRA_ARGS` to extend default args without overriding defaults

Open
#1,025 1 comment 1 reaction 0 assignees View on GitHub
expected: maybe someday good first ticket help wanted size: easy status: backlog touches: configuration why: functionality
Dominant language
Python
Stars
28.4k
Forks
1.6k
Avg merge
7h 54m
Merged PRs (30d)
21

Description

These `WGET_ARGS`, `CURL_ARGS`, etc. options let the user shoot themselves in the foot, silently. I think the documentation or the variables themselves should be changed to be more ergonomic.

Did you need to add a header to wget?
```py
WGET_ARGS=['--header=Accept-Language: en-US,en']
```
Look at this configuration, it looks inoffensive. Checking [the documention](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#wget_args), nothing ticks you off you're using it wrong. **But no.**
After this, your 🆆 button won't take you to the main html you archived, and wget archives things slighlty differently, silently.

Why? By setting WGET_ARGS you overwrote [vital settings](https://github.com/ArchiveBox/ArchiveBox/blob/03eb7e58758d8dcb85ed781e713fc083f8292264/archivebox/config.py#L159), as it turns out they're also stored in WGET_ARGS. The documentation doesn't tell you about this. This happened to me, I just happily overwrote the variable. When I should've written something like this:
```py
WGET_ARGS=['--header=Accept-Language: en-US,en;q=0.5', '--no-verbose', '--adjust-extension', '--convert-links', '--force-directories', '--backup-converted', '--span-hosts', '--no-parent', '-e', 'robots=off']
```

Proposal:
1. The documentation for `*_ARGS` should have a good warning or display the default value, so that users can suspect they're overwriting something.
2. Or, create `EXTRA_WGET_ARGS`, `EXTRA_CURL_ARGS`, and so on. These won't overwrite the now-considered low-level `*_ARGS`. `EXTRA_*_ARGS` shall be the more user-facing option and more promoted in the documentation.

Contributor guide

Open the contributing guide

Research direction

Start in archivebox/config.py at the WGET_ARGS defaults linked by the issue, then review the Configuration wiki section for WGET_ARGS. Define how the proposed EXTRA_*_ARGS variables relate to existing defaults for wget, curl, and SingleFile, and update the documentation so users know which settings preserve defaults; done means the new options extend rather than replace them.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.