apache / apache/datafusion-benchmarks

Incorrect "Fix bad UTF-8 char " step in Dockerfile actually corrupts the input

Open
#33 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
23
Forks
19
PR merge metrics
No merged PRs in 30d

Description

I spent a large amount of time confused about this and wanted to file an issue

I found it in the context of
- https://github.com/clflushopt/tpchgen-rs/pull/253

The Dockerfile in this repo implies that the DSGen data generator for TPCH has incorrect input:
https://github.com/apache/datafusion-benchmarks/blob/cb12c981e6608e0f2dcf919956ada8f1f1622d72/tpcds/Dockerfile#L17-L19

However, what that command actually does is corrupt the file

For example,t he originl data has a `CÔTE` in it: (\x43 \xc3 \x94 \x54 \x45 in utf8)

grep -n "IVOIRE" tpcds.dst | head -1 | xxd | head -3
echo "---"…)
⎿  00000000: 3634 393a 6164 6420 2822 43c3 9454 4520 649:add ("C..TE
00000010: 4427 4956 4f49 5245 223a 3129 3b0a D'IVOIRE":1);.

```
iconv -f ISO-8859-14 -t UTF-8
```

Tells `iconv` "treat the input as ISO-8859-14 single-byte data." But the file was already UTF-8. So iconv took each existing UTF-8 byte (e.g. C3 and 94) as if each were a separate Latin character, and re-encoded each one into UTF-8.

This results in

tpcds.dst (original) — already correct UTF-8:
- C3 94 = the valid UTF-8 encoding for Ô (U+00D4 LATIN CAPITAL LETTER O WITH CIRCUMFLEX)

tpcds.dst2 (after iconv) — corrupted (double-encoded):
- C3 83 C2 94 = two characters: Ã (U+00C3) + a control character (U+0094)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.