binary-husky / binary-husky/gpt_academic
[Feature]: 从arxiv上下载文件,配置国内访问加速,并增加进度条显示
- Dominant language
- Python
- Stars
- 71.4k
- Forks
- 8.3k
- PR merge metrics
- No merged PRs in 30d
Description
### Class | 类型
程序主体
### Feature Request | 功能请求
直接使用原始arxiv_download代码,经常出现从网站下载文件失败的状况,建议配置国内访问加速。比较简易的更改方式是在requests.get请求参数中增加headers,配置方法为:
```python
url = file_url
headers = {
'User-Agent': 'Lynx'
}
response = requests.get(url, headers=headers, stream=True)
```
另外,增加进度条可直观显示下载进度,可通过以下代码实现:
```python
total_size = int(response.headers.get('content-length', 0)) # 获取文件总大小
block_size = 1024 # 每次下载的块大小
# 使用 tqdm 创建进度条
with open(filename, 'wb') as file, tqdm(
desc=filename,
total=total_size,
unit='iB',
unit_scale=True,
unit_divisor=1024,
) as bar:
for data in response.iter_content(block_size):
file.write(data) # 写入文件
bar.update(len(data)) # 更新进度条
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the arxiv_download implementation and locate the requests.get call that retrieves the file. Check how dependencies are declared before adding the suggested request headers and tqdm-based progress reporting. Done means downloads show progress when a content length is available and still save the complete file successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100