apache / apache/cordova-plugin-file

readAsText function causes encoding error reading large files

Open
#284 14 comments 7 reactions 0 assignees View on GitHub
platform: android platform: browser platform: ios platform: windows support
Dominant language
JavaScript
Stars
744
Forks
754
Avg merge
3d 4h
Merged PRs (30d)
1

Description

# cordova-file-plugin encoding error bug report
## Abstract
The `readAsText` function of the cordova-file-plugin causes an encoding error when reading an UTF-8 file larger than 2 megabytes on platforms iOS, Android and Windows.

## Description
The error is caused by the native implementation of the [`readAsText`](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/src/ios/CDVFile.m#L866) function of the `cordova-file-plugin`. Files are cut into chunks at a predefined [size](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/www/FileReader.js#L52) and are immediately converted to UTF-8. UTF-8 encoded characters use 4 bytes and if the cut is exactly in one character with 4 bytes, an encoding error is thrown.
You can see the UTF-8 leading byte `e2` at the end of [data-chunk.dump file](./data-chunk.dump). The missing bytes are in the next chunk and if you convert the dump file back to text a filler character is shown (e.g. a `?`).

## Workaround
We implemented a workaround using the [`readAsArrayBuffer`](https://github.com/chauthai/cordova-file-plugin-bug-report/blob/628b8cd65b295042ee953d6241d8d84fe5e27565/index-array-buffer.html#L39) function of the `cordova-file-plugin` and converting the typed array to UTF-8 in JavaScript with [TextDecoder.decode()](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode).

## How to reproduce the `encoding error`
1. Check out [sampe-app](https://github.com/chauthai/cordova-file-plugin-bug-report)
1. Install dependencies
`npm i`
1. Initialize Cordova
`npm run init`
1. Deploy on platform
`npm run ios/android/windows`
1. Press `Fire Parser` in Demo App
1. Watch the logs
1. Set a breakpoint at the throw of [encoding error](https://github.com/apache/cordova-plugin-file/blob/d135cd0e006736ea60c9dec46590de0927d57b6b/src/ios/CDVFile.m#L872) in iOS
1. Look at the `data` variable

Related to #238

Contributor guide

Open the contributing guide

Research direction

Start with the linked native readAsText implementation in src/ios/CDVFile.m and the chunk-size logic in www/FileReader.js, then use the linked sample app and data-chunk.dump to reproduce the UTF-8 boundary failure. Done means reading large UTF-8 files through readAsText without an encoding error or replacement character on iOS, Android, and Windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, javascript, objective-c
Domain
mobile
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.