[feature] Enhance COPY INTO with multi-format support and advanced options
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
In production environments, bulk data import and export is a common and critical operation — loading datasets from external storage into Paimon tables, or exporting table data to files for downstream consumption. The `COPY INTO` statement provides a declarative SQL interface for these operations without requiring users to write custom ETL pipelines.
The initial `COPY INTO` implementation was recently introduced with basic CSV support. This umbrella issue tracks the effort to extend it with multi-format support and advanced options, aligning with the commonly adopted capabilities in the industry.
### Planned Features
We plan to implement the following capabilities as separate PRs:
#### Format Support
- [x] **JSON format** — Support `FILE_FORMAT = (TYPE = JSON)` for both import and export. Options: `MULTI_LINE`, `COMPRESSION`, `NULL_IF`, `EMPTY_FIELD_AS_NULL`. JSON import uses column-name matching by default (no positional dependency).
- [x] **Parquet format** — Support `FILE_FORMAT = (TYPE = PARQUET)` for both import and export. Leverages Spark's native schema inference. Options: `COMPRESSION`.
#### Error Handling
- [x] **ON_ERROR = CONTINUE / SKIP_FILE** — Currently only `ABORT_STATEMENT` is supported. `CONTINUE` skips bad rows and reports error counts per file. `SKIP_FILE` skips entire files that fail and reports per-file LOADED/LOAD_FAILED status.
#### Export Enhancement
- [ ] **FROM (SELECT ...) query export** — Currently export only supports `FROM table_name`. This adds support for arbitrary SQL queries as the data source for `COPY INTO `.
### Considered but deprioritized
- **MATCH_BY_COLUMN_NAME / ENABLE_SCHEMA_EVOLUTION** — Prototyped locally but deprioritized for now; out of scope for this issue.
### Proposed SQL Syntax
#### Import
```sql
COPY INTO table_name [(col1, col2, ...)]
FROM 'source_path'
FILE_FORMAT = (TYPE = CSV | JSON | PARQUET [, option = value, ...])
[PATTERN = 'regex']
[FORCE = TRUE | FALSE]
[ON_ERROR = ABORT_STATEMENT | CONTINUE | SKIP_FILE]
Export
COPY INTO 'target_path'
FROM { table_name | (SELECT ...) }
FILE_FORMAT = (TYPE = CSV | JSON | PARQUET [, option = value, ...])
[OVERWRITE = TRUE | FALSE]
```
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the existing COPY INTO SQL entry point and review how export currently accepts a table name as its source. Implement support for the proposed FROM (SELECT ...) form, then verify that arbitrary query results can be exported while preserving the existing format and overwrite options.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100