[Feature] Bash script to concurrently download from a list
- 主要言語
- Go
- スター
- 373
- フォーク
- 43
- PR マージ指標
- 30日以内にマージされた PR はありません
説明
## Feature Proposal
Hi, I've written a simple bash script that allows you to download multiple files from a list, using concurrency.
Essentially, the script requires a list of `n` URLs and will generate `n` processes that call `vimeo-dl`.
I just tried downloading ~50 files and it's worked flawlessly. Eventually, it could be integrated directly in the Go code.
## Bash script
```bash
#!/bin/bash
# Read the list of URLs from the file
urls=$(cat $1)
# Count the number of URLs
url_count=$(echo "$urls" | wc -l)
# User agent. You can change it as you wish
user_agent="Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36"
# Function to download using vimeo-dl. Add flags if needed.
download_video() {
local url=$1
vimeo-dl \
--combine \
-i "$url" \
--user-agent "$user_agent"
}
# Export the function so it can be used by parallel
export -f download_video
# Run the download_video function concurrently for each URL
echo "$urls" | xargs -n 1 -P "$url_count" -I {} bash -c 'download_video "$@"' _ {}
```
## Usage
1. Generate a `list.txt` file containing master.json urls.
2. Execute the script and pass the list via args:
`./script.sh list.txt`
3. Your videos will be exported to your pwd.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。