m-bain / m-bain/webvid

Is there a download limit imposed by the video source website?

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
686
Forks
40
PR merge metrics
No merged PRs in 30d

Description

I run this commad:
```
video2dataset --url_list="results_2M_train.csv" \
--input_format="csv" \
--output-format="webdataset" \
--output_folder="test" \
--url_col="contentUrl" \
--caption_col="name" \
--save_additional_columns='[videoid,page_idx,page_dir,duration]' \
--enable_wandb=False \
--config=default \
```
At first, the download process went smoothly, and I successfully downloaded 96 .tar files, totaling about 200GB. Then, error messages started appearing.
```
HTTPSConnectionPool(host='ak.picdn.net', port=443): Read timed out.
```
I switched to a different computer and attempted to download again, but encountered the same errors after downloading around 200GB.
Could this be due to a download limit imposed by the video source website?
How should I resolve this issue?
@m-bain

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository file or test is named; start by tracing the video2dataset download path and its handling of Read timed out errors. Compare the failure after roughly 200GB across the shown command and another run. Done means establishing whether the cause is source-side limiting or downloader behavior and documenting a verified resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.