HaxeFoundation / HaxeFoundation/haxe
Handling filesystems with non-Unicode filenames
- Dominant language
- Haxe
- Stars
- 6.9k
- Forks
- 715
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 11
Description
[Many filesystems](https://en.wikipedia.org/wiki/Comparison_of_file_systems) allow filenames in the format:
> Any byte except NUL, / *(ext4)*
or even:
> Any byte except : *(MFS)*
Both NTFS, ReFS (Windows) and HFS+ (macOS) impose a fairly reasonable limitation:
> Any Unicode except NUL, / *(ReFS)*
With an only-Unicode paths limitation, we can express paths as Unicode `String`s in Haxe and have nice `FileSystem` interfaces. Since we support Linux, however, there is a potential problem when running on e.g. ext4 and needing to read or write into a file with a name that is not a valid Unicode string.
*(Tested on RC.2, Ubuntu 14.04 on an ext4 filesystem.)* The behaviour atm is very inconsistent across targets:
```bash
$ mkdir invalid
$ cd invalid
$ touch `printf "\xc3\x28"` # (this creates a file with a Unicode-invalid filename)
```
```haxe
class Test {
public static function main():Void {
trace(sys.FileSystem.readDirectory("invalid")[0].charCodeAt(0));
}
}
```
On eval, `filename.charCodeAt(0) == 232`
On neko, `filename.charCodeAt(0) == 195`
On PHP, `filename.charCodeAt(0) == 104`
On Python, `filename.charCodeAt(0) == 56515` (AND it thinks the String is valid Unicode, i.e. Python sanitised it before Haxe got it)
On JS (node), `filename.charCodeAt(0) == 65533`
The proposed solution by @ncannasse was to have a scheme for "escaping" and "unescaping" paths between the filesystem and Haxe. Any invalid Unicode sequence would be replaced by a special Unicode character followed by the original byte turned into a Unicode point, i.e. `"" ` -> `""` (note that `U+007E` is not the same as the byte `7E`).
Contributor guide
Assessment
This issue has not been assessed yet.