ArchiveBox / ArchiveBox/abx-plugins
Bug: running pull (or update --resume via CLI) doesn't finish incomplete youtube-dl download
- Dominant language
- Python
- Stars
- 14
- Forks
- 3
- Avg merge
- 3h 12m
- Merged PRs (30d)
- 15
Description
#### Describe the bug
A YouTube video didn't complete its download before the timeout, so I have lots of files in the site's media directory representing subtitles, partial video files etc. but youtube-dl hasn't finished compiling them into one webm file. However, selecting the site and clicking "pull" in the web UI or running `archivebox update --resume ` doesn't appear to re-run youtube-dl to finish the video archival.
#### Steps to reproduce
1. Start archival of a YouTube video.
2. Cancel the youtube-dl step so that the download is only partial.
3. Attempt to resume download with web UI or CLI and note that it doesn't complete the video archival.
#### Screenshots or log output
Partially redacted output, showing "success" for the video in question despite it not finishing the archival.
```bash
$ docker compose run -e SAVE_MEDIA=1 archivebox update --resume 1673798553.228281
[i] [2023-02-19 16:45:22] ArchiveBox v0.6.2: archivebox update --resume 1673798553.228281
> /data
[▶] [2023-02-19 16:46:54] Starting archiving of 14 snapshots in index...
[redacted]
[√] [2023-02-19 16:47:00] ""
https://www.youtube.com/watch?v=
√ ./archive/1673798553.228281
268 files (310.7 MB) in 0:00:00s
[redacted]
[√] [2023-02-19 16:47:03] Update of 14 pages complete (9.03 sec)
- 12 links skipped
- 2 links updated
```
#### ArchiveBox version
```logs
ArchiveBox v0.6.2
Cpython Linux Linux-5.4.0-139-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep
[i] Dependency versions:
√ ARCHIVEBOX_BINARY v0.6.2 valid /usr/local/bin/archivebox
√ PYTHON_BINARY v3.9.5 valid /usr/local/bin/python3.9
√ DJANGO_BINARY v3.1.10 valid /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py
√ CURL_BINARY v7.64.0 valid /usr/bin/curl
√ WGET_BINARY v1.20.1 valid /usr/bin/wget
√ NODE_BINARY v15.14.0 valid /usr/bin/node
√ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file
√ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor
√ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js
- GIT_BINARY - disabled /usr/bin/git
√ YOUTUBEDL_BINARY v2021.04.26 valid /usr/local/bin/youtube-dl
√ CHROME_BINARY v90.0.4430.93 valid /usr/bin/chromium
√ RIPGREP_BINARY v0.10.0 valid /usr/bin/rg
[i] Source-code locations:
√ PACKAGE_DIR 22 files valid /app/archivebox
√ TEMPLATES_DIR 3 files valid /app/archivebox/templates
- CUSTOM_TEMPLATES_DIR - disabled
[i] Secrets locations:
- CHROME_USER_DATA_DIR - disabled
- COOKIES_FILE - disabled
[i] Data locations:
√ OUTPUT_DIR 5 files valid /data
√ SOURCES_DIR 71 files valid ./sources
√ LOGS_DIR 1 files valid ./logs
√ ARCHIVE_DIR 1258 files valid ./archive
√ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf
√ SQL_INDEX 9.6 MB valid ./index.sqlite3
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported `archivebox update --resume ` flow after cancelling a YouTube download, then trace the youtube-dl handling used by the CLI and web UI. Done means an incomplete download is detected and re-run so its partial files are compiled into the final webm, with the update no longer reporting a false success.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100