Array types break csv ingress/egress
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 153
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 125
Description
e.g.,
```sql
CREATE TABLE inputs
(
int_arr1d BIGINT ARRAY
);
CREATE VIEW outputs AS SELECT * FROM inputs;
```
will accept something like:
```
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/ingress/INPUTS?format=csv' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: application/json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'content-type: text/csv' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '0,4'
```
for input (note that those are supposedly two elements in the array:
which confuses then the retrieval:
```
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/egress/INPUTS?format=csv&query=quantiles&mode=snapshot' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'Content-Type: application/json' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '{"before":25,"after":25}'
```
will never send any data because on the backend it fails to serialize rows:
```
2023-08-07 21:28:42 ERROR [pipeline-0189b2c6-0135-7f9e-95fd-93f99592da61] encoder error on output endpoint 'api-snapshot-INPUTS-quantiles-4e0a9418-71f8-4753-a188-9e16a026bd95': CSV error: found record with 15 fields, but the previous record has 14 fields
```
So the problem seems to be that CSV parsing gets confused with arrays. We discussed as a potential solution, we need to define a standard for the CSV format to be e.g., strings of "[1,2,3]" for 1d arrays, or "[[1],[2,3]]" for 2d etc. (and any other future complex types for that matter). Then this can be recognized and parsed properly.
Contributor guide
Assessment
This issue has not been assessed yet.