feldera / feldera/feldera

Array types break csv ingress/egress

Open
#435 1 comment 0 reactions 1 assignee Claimed by @ryzhyk View on GitHub
bug Pipeline manager
Dominant language
Rust
Stars
2.1k
Forks
153
Avg merge
2d 12h
Merged PRs (30d)
125

Description

e.g.,

```sql
CREATE TABLE inputs
(
int_arr1d BIGINT ARRAY
);
CREATE VIEW outputs AS SELECT * FROM inputs;
```

will accept something like:

```
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/ingress/INPUTS?format=csv' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: application/json' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'content-type: text/csv' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '0,4'
```

for input (note that those are supposedly two elements in the array:

which confuses then the retrieval:

```
curl 'http://localhost:8080/v0/pipelines/0189b2c6-0135-7f9e-95fd-93f99592da61/egress/INPUTS?format=csv&query=quantiles&mode=snapshot' -X POST -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'Referer: http://localhost:3000/' -H 'Content-Type: application/json' -H 'Origin: http://localhost:3000' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: same-site' --data-raw '{"before":25,"after":25}'
```

will never send any data because on the backend it fails to serialize rows:

```
2023-08-07 21:28:42 ERROR [pipeline-0189b2c6-0135-7f9e-95fd-93f99592da61] encoder error on output endpoint 'api-snapshot-INPUTS-quantiles-4e0a9418-71f8-4753-a188-9e16a026bd95': CSV error: found record with 15 fields, but the previous record has 14 fields
```

So the problem seems to be that CSV parsing gets confused with arrays. We discussed as a potential solution, we need to define a standard for the CSV format to be e.g., strings of "[1,2,3]" for 1d arrays, or "[[1],[2,3]]" for 2d etc. (and any other future complex types for that matter). Then this can be recognized and parsed properly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.