apache / apache/cordova-plugin-file
readAsText function causes encoding error reading large files
- Dominant language
- JavaScript
- Stars
- 744
- Forks
- 754
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 1
Description
# cordova-file-plugin encoding error bug report
## Abstract
The `readAsText` function of the cordova-file-plugin causes an encoding error when reading an UTF-8 file larger than 2 megabytes on platforms iOS, Android and Windows.
## Description
The error is caused by the native implementation of the [`readAsText`](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/src/ios/CDVFile.m#L866) function of the `cordova-file-plugin`. Files are cut into chunks at a predefined [size](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/www/FileReader.js#L52) and are immediately converted to UTF-8. UTF-8 encoded characters use 4 bytes and if the cut is exactly in one character with 4 bytes, an encoding error is thrown.
You can see the UTF-8 leading byte `e2` at the end of [data-chunk.dump file](./data-chunk.dump). The missing bytes are in the next chunk and if you convert the dump file back to text a filler character is shown (e.g. a `?`).
## Workaround
We implemented a workaround using the [`readAsArrayBuffer`](https://github.com/chauthai/cordova-file-plugin-bug-report/blob/628b8cd65b295042ee953d6241d8d84fe5e27565/index-array-buffer.html#L39) function of the `cordova-file-plugin` and converting the typed array to UTF-8 in JavaScript with [TextDecoder.decode()](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode).
## How to reproduce the `encoding error`
1. Check out [sampe-app](https://github.com/chauthai/cordova-file-plugin-bug-report)
1. Install dependencies
`npm i`
1. Initialize Cordova
`npm run init`
1. Deploy on platform
`npm run ios/android/windows`
1. Press `Fire Parser` in Demo App
1. Watch the logs
1. Set a breakpoint at the throw of [encoding error](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/src/ios/CDVFile.m#L872) in iOS
1. Look at the `data` variable
Related to #238
Contributor guide
Research direction
Start with the linked native readAsText implementation in src/ios/CDVFile.m and the chunk-size logic in www/FileReader.js, then use the linked sample app and data-chunk.dump to reproduce the UTF-8 boundary failure. Done means reading large UTF-8 files through readAsText without an encoding error or replacement character on iOS, Android, and Windows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, javascript, objective-c
- Domain
- mobile
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100