apache / apache/cordova-plugin-file

Reading a small chunk of a very large file leads to OutOfMemory Exception (android)

Open
#388 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
JavaScript
Stars
744
Forks
754
Avg merge
3d 4h
Merged PRs (30d)
1

Description

# Bug Report

## Problem
Reading with `FileReader.readAsArrayBuffer` a chunk of a very large file (2Gb+) causes OutOfMemory on Android (not tested on iOS).

### What is expected to happen?
Reading a 5Mb chunk, using `FileReader.readAsArrayBuffer` on a sliced portion ( `FileEntry.slice(x,y)` ) of a very large file (2Gb+) should happen without any error, since the file is large, but the chunk is small.

### What does actually happen?
The read operation actually causes an OutOfMemory exception.

## Information
The final goal of this is to upload large files to our server. We do it chunking the file for many reason (upload time, resuming, memory problems, etc.). The whole file was read correctly, until it reached a certain point and the app crashed with OutOfMemory exception. Reading any part prior to that point seems to work fine.

I found some issues like this around the net, but none could help me.
I even tried applying [THIS](https://stackoverflow.com/questions/54946864/out-of-memory-error-when-calling-readasarraybuffer-method-on-filereader-of-the-c) fix but to no avail.

### Command or Code
I created a test project to test this, which you can download [HERE](https://www.dropbox.com/s/xo9rbm4xaf6tqff/outOfMemTest.zip?dl=0).
It is built in Ionic4 and uses this plugin (cordova-plugin-file) and cordova-plugin-filechooser to choose the file.

NOTE: To execute this project you need ionic4 and cordova installed on your computer.
To run it:

1. Download zip
2. Extract zip
3. open CMD, get into the project folder
4. run `npm install`
5. run `ionic cordova run android`
6. attach your android device or run an android emulator
7. run `ionic cordova run android`
8. Once in the project, tap the only button; a file chooser should open
9. choose a file larger then 2050Mb (the larger the better)
10. the app should crash; if you look ad `adb logcat` you should see an OutOfMemory error

Feel free to play with the `partStart` and `partEnd` const to try reading other parts of the file.

It reads the file from the provided URL using `resolveLocalFileSystemUrl,` then uses `slice` on the "file" object in the returned FileEntry. The sliced blob (wich is an IFile with adjusted start and end, AFAIK) is then passed to the `readAsArrayBuffer` function.

The following is the code that reads the file:

```
this.fileAPI.resolveLocalFilesystemUrl(filePath).then((fileEntry: FileEntry) => {

fileEntry.file((file) => {

const partStart = 2144337920;
const partEnd = 2149580800;

const reader = new FileReader();
const blob = file.slice(partStart, partEnd);

reader.onload = (event: any) => {
if (event.target.readyState === FileReader.DONE) {
console.log('READ COMPLETE');
console.log(event.target.result);
}
};

reader.readAsArrayBuffer(blob);

}, (error) => {
console.log('DEBUG - ERROR 3 ');
console.log(error);

});

}, (error) => {
console.log('DEBUG - ERROR 5 ');
console.log(error);

});
```

### Environment, Platform, Device
This was tested on an android emulator with these specs:

-Based on Pixel 2 device
-API level : 29 (Android Q)
-Graphics: Hardware - GLES 2.0
-Multi-Core CPU: 4
-RAM : 2048 Mb
-VM heap : 256 Mb
-Internal Storage: 10240 Mb
-SD card: Studio-managed 512 Mb

### Version information
Ionic:

Ionic CLI : 5.4.11
Ionic Framework : @ionic/angular 5.0.7
@angular-devkit/build-angular : 0.803.26
@angular-devkit/schematics : 8.3.26
@angular/cli : 8.3.26
@ionic/angular-toolkit : 2.2.0

Cordova:

Cordova CLI : 9.0.0 (cordova-lib@9.0.1)
Cordova Platforms : android 8.1.0
Cordova Plugins : cordova-plugin-ionic-keyboard 2.2.0, cordova-plugin-ionic-webview 4.1.3, (and 6 other plugins)

Utility:

cordova-res : 0.11.0
native-run : 0.2.7

System:

Android SDK Tools : 26.1.1
NodeJS : v10.13.0
npm : 6.4.1
OS : Windows 10

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the linked Ionic/Cordova test project, using the FileEntry.slice and FileReader.readAsArrayBuffer entry points with a file larger than 2050Mb. Check adb logcat for the Android OutOfMemory failure and compare different partStart and partEnd values. Done means a small sliced chunk can be read on Android without the app crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
android, javascript, node.js
Domain
mobile
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.