matrixorigin / matrixorigin/matrixone

[Feature Request]: file external table should support error mode

Open
#27,517 1 comment 0 reactions 1 assignee Claimed by @fengttt View on GitHub
kind/feature
Dominant language
Go
Stars
1.9k
Forks
311
Avg merge
1d 3h
Merged PRs (30d)
768

Description

### Is there an existing issue for the same feature request?

- [x] I have checked the existing issues.

### Is your feature request related to a problem?

```Markdown

```

### Describe the feature you'd like

When read from external table, add two builtin columns. Either csv, or jsonl,
if an error occurs, parsing/query should continue instead of fail.

```
__MO_FILE_LINE: This is the physical line number within the file, if the record is multiline (quoted field of CSV file), the beginning line of the record.
__MO_ERROR_MESSAGE: A short error message on why the parsing failed
__MO_ERROR_TEXT: The failed line, in text
```

Optimizer should perform column pruning first, and if query does not mention these columns, the behavior should be the same as existing code, this is, query will fail, and if query succeed there should be no performance penalty. If only __MO_FILE_LINE is used in the query, also fail the query if there is parsing error.

If at least one of the __MO_ERROR_MESSAGE or __MO_ERROR_TEXT columns is not pruned in the query, when parser sees an error, it should continue and set __MO_ERROR_XXX columns to null. If there is an error, set __MO_ERROR_XXX column and set all user columns to NULL. __MO_FILEPATH and __MO_FILE_LINE should always be set regardless of if there is an error.

### Describe implementation you've considered

_No response_

### Documentation, Adoption, Use Case, Migration Strategy

```Markdown

```

### Additional information

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.