Is there a download limit imposed by the video source website?
- Dominant language
- Python
- Stars
- 686
- Forks
- 40
- PR merge metrics
- No merged PRs in 30d
Description
I run this commad:
```
video2dataset --url_list="results_2M_train.csv" \
--input_format="csv" \
--output-format="webdataset" \
--output_folder="test" \
--url_col="contentUrl" \
--caption_col="name" \
--save_additional_columns='[videoid,page_idx,page_dir,duration]' \
--enable_wandb=False \
--config=default \
```
At first, the download process went smoothly, and I successfully downloaded 96 .tar files, totaling about 200GB. Then, error messages started appearing.
```
HTTPSConnectionPool(host='ak.picdn.net', port=443): Read timed out.
```
I switched to a different computer and attempted to download again, but encountered the same errors after downloading around 200GB.
Could this be due to a download limit imposed by the video source website?
How should I resolve this issue?
@m-bain
Contributor guide
No contributing guide indexed for this repository
Research direction
No repository file or test is named; start by tracing the video2dataset download path and its handling of Read timed out errors. Compare the failure after roughly 200GB across the shown command and another run. Done means establishing whether the cause is source-side limiting or downloader behavior and documenting a verified resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100