apache / apache/datafusion-benchmarks

Incorrect "Fix bad UTF-8 char " step in Dockerfile actually corrupts the input

Offen Anfängerfreundlich
#33 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
23
Forks
19
PR-Merge-Kennzahlen
Keine gemergten PRs in 30 T.

Beschreibung

I spent a large amount of time confused about this and wanted to file an issue

I found it in the context of
- https://github.com/clflushopt/tpchgen-rs/pull/253

The Dockerfile in this repo implies that the DSGen data generator for TPCH has incorrect input:
https://github.com/apache/datafusion-benchmarks/blob/cb12c981e6608e0f2dcf919956ada8f1f1622d72/tpcds/Dockerfile#L17-L19

However, what that command actually does is corrupt the file

For example,t he originl data has a `CÔTE` in it: (\x43 \xc3 \x94 \x54 \x45 in utf8)

grep -n "IVOIRE" tpcds.dst | head -1 | xxd | head -3
echo "---"…)
⎿  00000000: 3634 393a 6164 6420 2822 43c3 9454 4520 649:add ("C..TE
00000010: 4427 4956 4f49 5245 223a 3129 3b0a D'IVOIRE":1);.

```
iconv -f ISO-8859-14 -t UTF-8
```

Tells `iconv` "treat the input as ISO-8859-14 single-byte data." But the file was already UTF-8. So iconv took each existing UTF-8 byte (e.g. C3 and 94) as if each were a separate Latin character, and re-encoded each one into UTF-8.

This results in

tpcds.dst (original) — already correct UTF-8:
- C3 94 = the valid UTF-8 encoding for Ô (U+00D4 LATIN CAPITAL LETTER O WITH CIRCUMFLEX)

tpcds.dst2 (after iconv) — corrupted (double-encoded):
- C3 83 C2 94 = two characters: Ã (U+00C3) + a control character (U+0094)

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Rechercherichtung

Start with the tpcds/Dockerfile lines 17–19 in the linked datafusion-benchmarks revision and inspect the iconv command applied to tpcds.dst. Verify the source bytes are already UTF-8, then confirm the Docker build no longer double-encodes characters such as Ô. Done means the generated benchmark input preserves the original UTF-8 content.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
dockerfile
Bereich
build-system
Issue-Typ
Bug
Schwierigkeit
2/5
Geschätzter Aufwand
1-3 Stunden
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
68/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.