Kaggle / Kaggle/kaggle-sdk-python

Shouldn't API client pass stream = True to the requests when downloading datasets?

Aperta
#10 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
10
Fork
7
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Linking to the issue of the same name in kaggle-api:

https://github.com/Kaggle/kaggle-api/issues/754

I'm raising it here since the problematic code is now maintained in kagglesdk ([KaggleHttpClient.call](https://github.com/Kaggle/kagglesdk/blob/main/kagglesdk/kaggle_http_client.py#L73) is where I think this could be fixed).

TL;DR Not using `stream=True` in `requests` is causing entire datasets to be materialized in memory which makes it impossible to download anything of even a modest size.

I refer to the linked issue for more details but happy to expand the conversation here.

Tagging @leoauri and @i-aki-y for their visibility.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Direzione di ricerca

Start in kagglesdk/kaggle_http_client.py at KaggleHttpClient.call, then read the linked kaggle-api issue for the dataset-download details. Confirm the client’s download behavior for larger datasets and verify that responses are handled without materializing the entire dataset in memory.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python
Ambito
api, backend
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
45/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.