4paradigm / 4paradigm/OpenMLDB
load results are different in two load_mode
- Dominant language
- C++
- Stars
- 1.7k
- Forks
- 331
- Avg merge
- 12d 12h
- Merged PRs (30d)
- 1
Description
version v0.7.1
The root cause is the way we read csv. cluster load_mode is spark, local load_mode is self-determined in https://github.com/4paradigm/OpenMLDB/blob/09540fe7f7efe36b0a0f515aa6f530e82220bb31/src/sdk/sql_cluster_router.cc#L2968
### blank value
src csv:
```
c1,c2
,
```
cluster load(online&offline) result is two `null`, local load result is two empty strings.
- [ ] local mode should be fixed later.
### quote
src csv:
```
c1,c2
,
"",""
```
load data to online in cluster load_mode, select * result is:
```
------ ------
c1 c2
------ ------
NULL NULL
------ ------
2 rows in set
```
It has an empty row, but unclear. Anyway, the first row `,` is `null, null`, and the second row `"",""` is empty string.
load data to online in local load_mode, select * result is:
```
---- ----
c1 c2
---- ----
"" ""
---- ----
2 rows in set
```
The first row `,` is empty string, and the second row is 2 double quotation marks `"",""`.
- [x] In spark 3.2.1, csv default quote is `"`, https://spark.apache.org/docs/3.2.1/sql-data-sources-csv.html. It's changed, so we should let default quote is `\0` or set local load_mode default quote to `"`(local load_mode can't do escape now).
Default quote is `"`.
- [ ] local mode can't be consistent with cluster mode, document it, fix later.
Contributor guide
Assessment
This issue has not been assessed yet.