apache / apache/cordova-plugin-file
FileWriter fails during FileRead step for large documents amidst a high volume of previous downloads on iOS
- Dominant language
- JavaScript
- Stars
- 744
- Forks
- 754
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 1
Description
The problem as the user sees it:
1. User has long list of documents they need to download to iOS device (200+)
2. User starts the download, with each file downloaded in succession.
3. At the end of the download queue, they discover that one of the files fails (and its always one of two specific files that are 25MB+)
4. They retry the job (which only downloads the failed document) and it succeeds
What I'm seeing as a developer:
1. My app pulls down the document as a blob
2. The app calls `this.file.writeFile(directoryPath, fileName, blob, {replace: true})`
3. For the file that fails, I see the error pop out as
```
{"type":"error","bubbles":false,"cancelBubble":false,"cancelable":false,"lengthComputable":false,"loaded":0,"total":0,"target":{"fileName":"","length":0,"localURL":"cdvfile://localhost/persistent/downloaded-assets/9ce34f8a-6201-4023-9f5b-de6133bd5699/{{redacted}}","position":0,"readyState":2,"result":null,"error":{},"onwritestart":null,"onprogress":null,"onwriteend":null,"onabort":null}}
```
Which is not super helpful.
My debugging has led me to observe that the download fails in FileReader.js (which is called as part of the FileWriter step) with the line `return this._realReader.readAsArrayBuffer(file);` https://github.com/apache/cordova-plugin-file/blob/4a92bbbea755aa9e5bf8cfd160fb9da1cd3287cd/www/FileReader.js#L287
This is problematic because _realReader is set as `this._realReader = origFileReader ? new origFileReader() : {};`, and I have no idea how to access that for debugging.
To make matters worse, when I put a debugging statement here: https://github.com/apache/cordova-plugin-file/blob/4a92bbbea755aa9e5bf8cfd160fb9da1cd3287cd/www/FileWriter.js#L125, the error just prints out as an empty hash, telling me nothing.
Do you have any idea what might be going on here? Is it a memory issue on iOS? Does Cordova timeout over multiple repeated requests?
Something to note, the full list download works fine every time on the iOS xCode simulator. This is leading me to believe it might be a memory issue, but I'm not sure.
EDIT: I now see that the error is happening in the Ionic plugin (though this still might be an issue with the way Blobs are maintained in a Cordova environment)
I put some logging into the Ionic File plugin `writeFileInChunks` method. It became clear that the plugin, for whatever reason, error'd during a `slice` step.
```
private writeFileInChunks(writer: FileWriter, file: Blob) {
console.log('SIZE OF FILE AT START', file.size);
const BLOCK_SIZE = 1024 * 1024;
let writtenSize = 0;
function writeNextChunk() {
const size = Math.min(BLOCK_SIZE, file.size - writtenSize);
console.log('CALCULATED SIZE:', size);
const chunk = file.slice(writtenSize, writtenSize + size);
console.log('SIZE OF CHUNK TO WRITE', chunk.size)
writtenSize += size;
writer.write(chunk);
}
return getPromise((resolve, reject) => {
writer.onerror = reject as (event: ProgressEvent) => void;
writer.onwrite = () => {
if (writtenSize < file.size) {
writeNextChunk();
} else {
resolve();
}
};
writeNextChunk();
});
}
```
The output for the failed document:
```
SIZE OF FILE AT START: 34012899
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 0
```
On retry:
```
SIZE OF FILE AT START: 34012899
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 1048576
CALCULATED SIZE: 1048576
SIZE OF CHUNK TO WRITE: 1048576
...
...
...
CALCULATED SIZE: 458467
SIZE OF CHUNK TO WRITE: 458467
```
So for whatever reason, after a large number of previous downloads, that file.slice step results in an empty/corrupted blob.
Any ideas on how to correct this?
Ran another test with some expanded logging:
```
private writeFileInChunks(writer: FileWriter, file: Blob) {
...
function writeNextChunk() {
const size = Math.min(BLOCK_SIZE, file.size - writtenSize);
console.log('CALCULATED SIZE:', size);
console.log('WRITTEN SIZE', writtenSize);
console.log('SUMS TO:', writtenSize + size)
console.log('FILE SIZE BEFORE SLICE:', file.size);
const chunk = file.slice(writtenSize, writtenSize + size);
console.log('SIZE OF CHUNK TO WRITE', chunk.size);
writtenSize += size;
writer.write(chunk);
}
...
...
}
```
Output came to:
```
CALCULATED SIZE: 1048576
WRITTEN SIZE: 0
SUMS TO: 1048576
FILE SIZE BEFORE SLICE: 34012899
SIZE OF CHUNK TO WRITE 0
```
Further confirming the issue
Contributor guide
Research direction
Start with www/FileReader.js at the _realReader.readAsArrayBuffer call and www/FileWriter.js around line 125, then trace the Ionic plugin's writeFileInChunks method and its file.slice call. Reproduce the large-document failure on an iOS device after many downloads and compare it with the simulator and retry behavior. Done means the affected large files produce valid chunks and complete the download reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ios, javascript
- Domain
- mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100