ArchiveBox / ArchiveBox/abx-plugins

Bug: running pull (or update --resume via CLI) doesn't finish incomplete youtube-dl download

Open
#52 7 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
14
Forks
3
Avg merge
3h 12m
Merged PRs (30d)
15

Description

#### Describe the bug

A YouTube video didn't complete its download before the timeout, so I have lots of files in the site's media directory representing subtitles, partial video files etc. but youtube-dl hasn't finished compiling them into one webm file. However, selecting the site and clicking "pull" in the web UI or running `archivebox update --resume ` doesn't appear to re-run youtube-dl to finish the video archival.

#### Steps to reproduce

1. Start archival of a YouTube video.
2. Cancel the youtube-dl step so that the download is only partial.
3. Attempt to resume download with web UI or CLI and note that it doesn't complete the video archival.

#### Screenshots or log output

Partially redacted output, showing "success" for the video in question despite it not finishing the archival.

```bash
$ docker compose run -e SAVE_MEDIA=1 archivebox update --resume 1673798553.228281
[i] [2023-02-19 16:45:22] ArchiveBox v0.6.2: archivebox update --resume 1673798553.228281
> /data

[▶] [2023-02-19 16:46:54] Starting archiving of 14 snapshots in index...

[redacted]

[√] [2023-02-19 16:47:00] ""
https://www.youtube.com/watch?v=
√ ./archive/1673798553.228281
268 files (310.7 MB) in 0:00:00s

[redacted]

[√] [2023-02-19 16:47:03] Update of 14 pages complete (9.03 sec)
- 12 links skipped
- 2 links updated
```

#### ArchiveBox version

```logs
ArchiveBox v0.6.2
Cpython Linux Linux-5.4.0-139-generic-x86_64-with-glibc2.28 x86_64
IN_DOCKER=True DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
√ ARCHIVEBOX_BINARY v0.6.2 valid /usr/local/bin/archivebox
√ PYTHON_BINARY v3.9.5 valid /usr/local/bin/python3.9
√ DJANGO_BINARY v3.1.10 valid /usr/local/lib/python3.9/site-packages/django/bin/django-admin.py
√ CURL_BINARY v7.64.0 valid /usr/bin/curl
√ WGET_BINARY v1.20.1 valid /usr/bin/wget
√ NODE_BINARY v15.14.0 valid /usr/bin/node
√ SINGLEFILE_BINARY v0.3.16 valid /node/node_modules/single-file/cli/single-file
√ READABILITY_BINARY v0.0.2 valid /node/node_modules/readability-extractor/readability-extractor
√ MERCURY_BINARY v1.0.0 valid /node/node_modules/@postlight/mercury-parser/cli.js
- GIT_BINARY - disabled /usr/bin/git
√ YOUTUBEDL_BINARY v2021.04.26 valid /usr/local/bin/youtube-dl
√ CHROME_BINARY v90.0.4430.93 valid /usr/bin/chromium
√ RIPGREP_BINARY v0.10.0 valid /usr/bin/rg

[i] Source-code locations:
√ PACKAGE_DIR 22 files valid /app/archivebox
√ TEMPLATES_DIR 3 files valid /app/archivebox/templates
- CUSTOM_TEMPLATES_DIR - disabled

[i] Secrets locations:
- CHROME_USER_DATA_DIR - disabled
- COOKIES_FILE - disabled

[i] Data locations:
√ OUTPUT_DIR 5 files valid /data
√ SOURCES_DIR 71 files valid ./sources
√ LOGS_DIR 1 files valid ./logs
√ ARCHIVE_DIR 1258 files valid ./archive
√ CONFIG_FILE 81.0 Bytes valid ./ArchiveBox.conf
√ SQL_INDEX 9.6 MB valid ./index.sqlite3
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the reported `archivebox update --resume ` flow after cancelling a YouTube download, then trace the youtube-dl handling used by the CLI and web UI. Done means an incomplete download is detected and re-run so its partial files are compiled into the final webm, with the update no longer reporting a false success.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.