HaxeFoundation / HaxeFoundation/haxe

Handling filesystems with non-Unicode filenames

Open
#8,134 8 comments 0 reactions 0 assignees View on GitHub
discussion unicode
Dominant language
Haxe
Stars
6.9k
Forks
715
Avg merge
2d 2h
Merged PRs (30d)
11

Description

[Many filesystems](https://en.wikipedia.org/wiki/Comparison_of_file_systems) allow filenames in the format:

> Any byte except NUL, / *(ext4)*

or even:

> Any byte except : *(MFS)*

Both NTFS, ReFS (Windows) and HFS+ (macOS) impose a fairly reasonable limitation:

> Any Unicode except NUL, / *(ReFS)*

With an only-Unicode paths limitation, we can express paths as Unicode `String`s in Haxe and have nice `FileSystem` interfaces. Since we support Linux, however, there is a potential problem when running on e.g. ext4 and needing to read or write into a file with a name that is not a valid Unicode string.

*(Tested on RC.2, Ubuntu 14.04 on an ext4 filesystem.)* The behaviour atm is very inconsistent across targets:

```bash
$ mkdir invalid
$ cd invalid
$ touch `printf "\xc3\x28"` # (this creates a file with a Unicode-invalid filename)
```

```haxe
class Test {
public static function main():Void {
trace(sys.FileSystem.readDirectory("invalid")[0].charCodeAt(0));
}
}
```

On eval, `filename.charCodeAt(0) == 232`
On neko, `filename.charCodeAt(0) == 195`
On PHP, `filename.charCodeAt(0) == 104`
On Python, `filename.charCodeAt(0) == 56515` (AND it thinks the String is valid Unicode, i.e. Python sanitised it before Haxe got it)
On JS (node), `filename.charCodeAt(0) == 65533`

The proposed solution by @ncannasse was to have a scheme for "escaping" and "unescaping" paths between the filesystem and Haxe. Any invalid Unicode sequence would be replaced by a special Unicode character followed by the original byte turned into a Unicode point, i.e. `"" ` -> `""` (note that `U+007E` is not the same as the byte `7E`).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.