Kaggle / Kaggle/kaggle-sdk-python
Shouldn't API client pass stream = True to the requests when downloading datasets?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 10
- Forks
- 7
- PR merge metrics
- No merged PRs in 30d
Description
Linking to the issue of the same name in kaggle-api:
https://github.com/Kaggle/kaggle-api/issues/754
I'm raising it here since the problematic code is now maintained in kagglesdk (KaggleHttpClient.call is where I think this could be fixed).
TL;DR Not using stream=True in requests is causing entire datasets to be materialized in memory which makes it impossible to download anything of even a modest size.
I refer to the linked issue for more details but happy to expand the conversation here.
Tagging @leoauri and @i-aki-y for their visibility.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in kagglesdk/kaggle_http_client.py at KaggleHttpClient.call, then read the linked kaggle-api issue for the dataset-download details. Confirm the client’s download behavior for larger datasets and verify that responses are handled without materializing the entire dataset in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100