react / react/react-native

Fetching large json data results in OutOfMemoryError

Ouverte
#32,134 22 commentaires 4 réactions 0 personnes assignées Voir sur GitHub

Personne n'a encore pris cette issue.

🌐Networking Issue: Author Provided Repro
Langage dominant
C++
Étoiles
127k
Forks
25.3k
Merge moyen
1 j 23 h
PR mergées (30 j)
4

Description

Description

Using fetch to get several Megabytes (currently happens for me at around 80Mb) of JSON data causes Android to panic and throw an OutOfMemoryError. This is due to the fact that the whole response is being read as bytes, quickly filling up the heap.

Stacktrace:

09-01 21:32:21.035  5480  5555 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate a 98836368 byte allocation with 25165824 free bytes and 88MB until OOM, target footprint 133385952, growth limit 201326592
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at okio.Buffer.readByteArray(Buffer.kt:1429)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at okio.Buffer.readByteArray(Buffer.kt:1424)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at okio.RealBufferedSource.readByteArray(RealBufferedSource.kt:238)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at okhttp3.ResponseBody.bytes(ResponseBody.kt:124)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at com.facebook.react.modules.blob.BlobModule$4.toResponseData(BlobModule.java:134)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at com.facebook.react.modules.network.NetworkingModule$2.onResponse(NetworkingModule.java:512)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
09-01 21:32:21.035  5480  5555 E AndroidRuntime: 	at java.lang.Thread.run(Thread.java:923)

React Native version:

System:
    OS: Linux 5.13 Solus 4.3
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Memory: 1.39 GB / 15.52 GB
    Shell: 5.1.8 - /bin/bash
  Binaries:
    Node: 14.17.5 - /usr/bin/node
    Yarn: 1.22.10 - /usr/bin/yarn
    npm: 6.14.14 - /usr/bin/npm
    Watchman: Not Found
  SDKs:
    Android SDK: Not Found
  IDEs:
    Android Studio: Not Found
  Languages:
    Java: 1.8.0_302-solus - /usr/lib64/openjdk-8/bin/javac
  npmPackages:
    @react-native-community/cli: Not Found
    react: 17.0.2 => 17.0.2 
    react-native: 0.65.1 => 0.65.1 
  npmGlobalPackages:
    *react-native*: Not Found

Steps To Reproduce

I created a test repository that shows the aforementioned behaviour. It also includes a 100Mb JSON file for testing, that you can serve locally or access via GitHub directly.
Steps are as follows

  1. Tap on the load data button
  2. Wait for the app to crash

Expected Results

Getting an out of memory error shouldn't really happen with this size of data in my opinion. Sure 100Mb sounds a lot at first, but in enterprise-grade apps this is probably a common scenario. In any case I think there should be a possibility to dynamically switch to streaming the response since okhttp offers bytestream and charstream as well. I'm not that well versed in Java but it would probably help reduce all of the 150Mb landing on the heap at once?

Snack, code example, screenshot, or link to a repository:

https://github.com/curtisy1/ReactNativeFetchRepro

Guide de contribution

Ouvrir le guide de contribution

Par où commencer

  1. Lisez l'issue en entier, puis le guide de contribution du projet.
  2. Signalez en commentaire que vous la prenez — cela évite que deux personnes fassent le même travail.
  3. Forkez le dépôt et travaillez sur une branche.
  4. Ouvrez une pull request qui référence le numéro de l'issue.

Piste de recherche

Commencez par examiner la gestion des réponses Android présentée dans BlobModule.java et NetworkingModule.java, puis exécutez le dépôt ReactNativeFetchRepro lié avec son fichier data.json de 100MB. Comparez le chemin fetch actuel avec les références à OkHttp ResponseBody byteStream et charStream dans l’issue ; le travail est terminé lorsque les grandes réponses JSON ne déclenchent plus l’OutOfMemoryError signalé.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
android, java, react-native
Domaine
mobile, networking
Type d'issue
Bug
Difficulté
4/5
Temps estimé
3-5 jours
Activité
À l'abandon
Clarté
Plutôt claire
Accessibilité débutants
42/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.