4paradigm / 4paradigm/OpenMLDB

load results are different in two load_mode

Offen
#3,015 0 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @vagetablechicken Auf GitHub ansehen
batch-engine bug high-priority storage-engine
Vorherrschende Sprache
C++
Sterne
1.7k
Forks
331
Ø Merge
12 T. 12 Std.
Gemergte PRs (30 T.)
1

Beschreibung

version v0.7.1

The root cause is the way we read csv. cluster load_mode is spark, local load_mode is self-determined in https://github.com/4paradigm/OpenMLDB/blob/09540fe7f7efe36b0a0f515aa6f530e82220bb31/src/sdk/sql_cluster_router.cc#L2968

### blank value
src csv:
```
c1,c2
,
```
cluster load(online&offline) result is two `null`, local load result is two empty strings.

- [ ] local mode should be fixed later.

### quote
src csv:
```
c1,c2
,
"",""
```
load data to online in cluster load_mode, select * result is:
```
------ ------
c1 c2
------ ------
NULL NULL
------ ------

2 rows in set
```
It has an empty row, but unclear. Anyway, the first row `,` is `null, null`, and the second row `"",""` is empty string.

load data to online in local load_mode, select * result is:
```
---- ----
c1 c2
---- ----
"" ""
---- ----

2 rows in set
```
The first row `,` is empty string, and the second row is 2 double quotation marks `"",""`.

- [x] In spark 3.2.1, csv default quote is `"`, https://spark.apache.org/docs/3.2.1/sql-data-sources-csv.html. It's changed, so we should let default quote is `\0` or set local load_mode default quote to `"`(local load_mode can't do escape now).
Default quote is `"`.

- [ ] local mode can't be consistent with cluster mode, document it, fix later.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

The issue is in sql_cluster_router.cc line 2968 where load_mode is determined. Examine the CSV reading logic for cluster (spark) vs local modes, focusing on handling of blank values and quote characters. Test with the provided CSV examples to see the differing outputs. The fix involves aligning local mode's quote handling with Spark's defaults or documenting the inconsistency.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
spark
Bereich
data-engineering, databases
Issue-Typ
Bug
Schwierigkeit
4/5
Geschätzter Aufwand
3-5 Tage
Aktivitätsstatus
Veraltet
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
45/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.