bazelbuild / bazelbuild/rules_pkg

Can we make UTF-8 handling/tests better on macOS & Windows?

Open
#491 1 comment 0 reactions 0 assignees View on GitHub
P3
Dominant language
Starlark
Stars
253
Forks
221
Avg merge
10h 33m
Merged PRs (30d)
1

Description

Currently we disable //tests/mappings:utf8_manifest_test on windows.
It is enabled for macOS, but fails for me when building on a case sensitive file system.
The problem for macOS is related to handling of some characters in paths. macOS uses an alternate encoding, so our expected paths vary from what we get. You can see the difference in our normalized encoding of "sübdir/2-λ". On macos, instead of getting a single glyph, you get an encoding of 'u' followed by a "put an umlat on the previous character.

```
$ cat -v tests/mappings/utf8_manifest.golden
[
[0,"1-a","tests/testdata/utf8/1-a","0644",null,null],
[0,"2-λ","tests/testdata/utf8/2-λ","0644",null,null],
[0,"3-�M-^V","tests/testdata/utf8/3-�M-^V","0644",null,null],
[0,"BUILD","tests/testdata/utf8/BUILD","0644",null,null],
[0,"sübdir/2-λ","tests/testdata/utf8/sübdir/2-λ","0644",null,null],
[0,"sübdir/hello","tests/testdata/utf8/sübdir/hello","0644",null,null]
]
$ cat -v bazel-bin/tests/mappings/utf8.manifest
[
[0,"1-a","tests/testdata/utf8/1-a","0644",null,null],
[0,"2-λ","tests/testdata/utf8/2-λ","0644",null,null],
[0,"3-�M-^V","tests/testdata/utf8/3-�M-^V","0644",null,null],
[0,"BUILD","tests/testdata/utf8/BUILD","0644",null,null],
[0,"su�M-^Hbdir/2-λ","tests/testdata/utf8/su�M-^Hbdir/2-λ","0644",null,null],
[0,"su�M-^Hbdir/hello","tests/testdata/utf8/su�M-^Hbdir/hello","0644",null,null]
]
```

This may need to be fixed in Bazel rather than here, I am just gathering data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.