matrixorigin / matrixorigin/matrixone
[Feature Request]: file external table should support error mode
- Dominant language
- Go
- Stars
- 1.9k
- Forks
- 311
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 768
Description
### Is there an existing issue for the same feature request?
- [x] I have checked the existing issues.
### Is your feature request related to a problem?
```Markdown
```
### Describe the feature you'd like
When read from external table, add two builtin columns. Either csv, or jsonl,
if an error occurs, parsing/query should continue instead of fail.
```
__MO_FILE_LINE: This is the physical line number within the file, if the record is multiline (quoted field of CSV file), the beginning line of the record.
__MO_ERROR_MESSAGE: A short error message on why the parsing failed
__MO_ERROR_TEXT: The failed line, in text
```
Optimizer should perform column pruning first, and if query does not mention these columns, the behavior should be the same as existing code, this is, query will fail, and if query succeed there should be no performance penalty. If only __MO_FILE_LINE is used in the query, also fail the query if there is parsing error.
If at least one of the __MO_ERROR_MESSAGE or __MO_ERROR_TEXT columns is not pruned in the query, when parser sees an error, it should continue and set __MO_ERROR_XXX columns to null. If there is an error, set __MO_ERROR_XXX column and set all user columns to NULL. __MO_FILEPATH and __MO_FILE_LINE should always be set regardless of if there is an error.
### Describe implementation you've considered
_No response_
### Documentation, Adoption, Use Case, Migration Strategy
```Markdown
```
### Additional information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.