Fetching large json data results in OutOfMemoryError
Nessuno ha ancora preso questa issue.
- Lingua principale
- C++
- Stelle
- 127k
- Fork
- 25.3k
- Merge medio
- 1g 23h
- PR unite (30g)
- 4
Descrizione
Description
Using fetch to get several Megabytes (currently happens for me at around 80Mb) of JSON data causes Android to panic and throw an OutOfMemoryError. This is due to the fact that the whole response is being read as bytes, quickly filling up the heap.
Stacktrace:
09-01 21:32:21.035 5480 5555 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate a 98836368 byte allocation with 25165824 free bytes and 88MB until OOM, target footprint 133385952, growth limit 201326592
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at okio.Buffer.readByteArray(Buffer.kt:1429)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at okio.Buffer.readByteArray(Buffer.kt:1424)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at okio.RealBufferedSource.readByteArray(RealBufferedSource.kt:238)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at okhttp3.ResponseBody.bytes(ResponseBody.kt:124)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at com.facebook.react.modules.blob.BlobModule$4.toResponseData(BlobModule.java:134)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at com.facebook.react.modules.network.NetworkingModule$2.onResponse(NetworkingModule.java:512)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
09-01 21:32:21.035 5480 5555 E AndroidRuntime: at java.lang.Thread.run(Thread.java:923)
React Native version:
System:
OS: Linux 5.13 Solus 4.3
CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
Memory: 1.39 GB / 15.52 GB
Shell: 5.1.8 - /bin/bash
Binaries:
Node: 14.17.5 - /usr/bin/node
Yarn: 1.22.10 - /usr/bin/yarn
npm: 6.14.14 - /usr/bin/npm
Watchman: Not Found
SDKs:
Android SDK: Not Found
IDEs:
Android Studio: Not Found
Languages:
Java: 1.8.0_302-solus - /usr/lib64/openjdk-8/bin/javac
npmPackages:
@react-native-community/cli: Not Found
react: 17.0.2 => 17.0.2
react-native: 0.65.1 => 0.65.1
npmGlobalPackages:
*react-native*: Not Found
Steps To Reproduce
I created a test repository that shows the aforementioned behaviour. It also includes a 100Mb JSON file for testing, that you can serve locally or access via GitHub directly.
Steps are as follows
- Tap on the load data button
- Wait for the app to crash
Expected Results
Getting an out of memory error shouldn't really happen with this size of data in my opinion. Sure 100Mb sounds a lot at first, but in enterprise-grade apps this is probably a common scenario. In any case I think there should be a possibility to dynamically switch to streaming the response since okhttp offers bytestream and charstream as well. I'm not that well versed in Java but it would probably help reduce all of the 150Mb landing on the heap at once?
Snack, code example, screenshot, or link to a repository:
Guida per i contributori
Apri la guida per i contributori
Come iniziare
- Leggi tutta la issue e poi la guida ai contributi del progetto.
- Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
- Fai un fork del repository e lavora su un branch.
- Apri una pull request che faccia riferimento al numero della issue.
Direzione di ricerca
Inizia dalla gestione delle risposte Android mostrata in BlobModule.java e NetworkingModule.java, quindi esegui il repository ReactNativeFetchRepro collegato con il suo file data.json da 100MB. Confronta il percorso fetch attuale con i riferimenti a OkHttp ResponseBody byteStream e charStream nell’issue; il lavoro è completato quando le risposte JSON di grandi dimensioni non causano più l’OutOfMemoryError segnalato.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- android, java, react-native
- Ambito
- mobile, networking
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 42/100