Kaggle / Kaggle/kaggle-sdk-python

Shouldn't API client pass stream = True to the requests when downloading datasets?

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
7
PR merge metrics
No merged PRs in 30d

Description

Linking to the issue of the same name in kaggle-api:

https://github.com/Kaggle/kaggle-api/issues/754

I'm raising it here since the problematic code is now maintained in kagglesdk ([KaggleHttpClient.call](https://github.com/Kaggle/kagglesdk/blob/main/kagglesdk/kaggle_http_client.py#L73) is where I think this could be fixed).

TL;DR Not using `stream=True` in `requests` is causing entire datasets to be materialized in memory which makes it impossible to download anything of even a modest size.

I refer to the linked issue for more details but happy to expand the conversation here.

Tagging @leoauri and @i-aki-y for their visibility.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.