Kaggle / Kaggle/kaggle-sdk-python
Shouldn't API client pass stream = True to the requests when downloading datasets?
- Lingua principale
- Python
- Stelle
- 10
- Fork
- 7
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
Linking to the issue of the same name in kaggle-api:
https://github.com/Kaggle/kaggle-api/issues/754
I'm raising it here since the problematic code is now maintained in kagglesdk ([KaggleHttpClient.call](https://github.com/Kaggle/kagglesdk/blob/main/kagglesdk/kaggle_http_client.py#L73) is where I think this could be fixed).
TL;DR Not using `stream=True` in `requests` is causing entire datasets to be materialized in memory which makes it impossible to download anything of even a modest size.
I refer to the linked issue for more details but happy to expand the conversation here.
Tagging @leoauri and @i-aki-y for their visibility.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Direzione di ricerca
Start in kagglesdk/kaggle_http_client.py at KaggleHttpClient.call, then read the linked kaggle-api issue for the dataset-download details. Confirm the client’s download behavior for larger datasets and verify that responses are handled without materializing the entire dataset in memory.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- python
- Ambito
- api, backend
- Tipo di issue
- Bug
- Difficoltà
- 3/5
- Tempo stimato
- 1-2 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 45/100