apache / apache/paimon

[feature] Enhance COPY INTO with multi-format support and advanced options

Open
#8,005 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

In production environments, bulk data import and export is a common and critical operation — loading datasets from external storage into Paimon tables, or exporting table data to files for downstream consumption. The `COPY INTO` statement provides a declarative SQL interface for these operations without requiring users to write custom ETL pipelines.

The initial `COPY INTO` implementation was recently introduced with basic CSV support. This umbrella issue tracks the effort to extend it with multi-format support and advanced options, aligning with the commonly adopted capabilities in the industry.

### Planned Features

We plan to implement the following capabilities as separate PRs:

#### Format Support
- [x] **JSON format** — Support `FILE_FORMAT = (TYPE = JSON)` for both import and export. Options: `MULTI_LINE`, `COMPRESSION`, `NULL_IF`, `EMPTY_FIELD_AS_NULL`. JSON import uses column-name matching by default (no positional dependency).
- [x] **Parquet format** — Support `FILE_FORMAT = (TYPE = PARQUET)` for both import and export. Leverages Spark's native schema inference. Options: `COMPRESSION`.

#### Error Handling
- [x] **ON_ERROR = CONTINUE / SKIP_FILE** — Currently only `ABORT_STATEMENT` is supported. `CONTINUE` skips bad rows and reports error counts per file. `SKIP_FILE` skips entire files that fail and reports per-file LOADED/LOAD_FAILED status.

#### Export Enhancement
- [ ] **FROM (SELECT ...) query export** — Currently export only supports `FROM table_name`. This adds support for arbitrary SQL queries as the data source for `COPY INTO `.


### Considered but deprioritized

- **MATCH_BY_COLUMN_NAME / ENABLE_SCHEMA_EVOLUTION** — Prototyped locally but deprioritized for now; out of scope for this issue.

### Proposed SQL Syntax

#### Import
```sql
COPY INTO table_name [(col1, col2, ...)]
FROM 'source_path'
FILE_FORMAT = (TYPE = CSV | JSON | PARQUET [, option = value, ...])
[PATTERN = 'regex']
[FORCE = TRUE | FALSE]
[ON_ERROR = ABORT_STATEMENT | CONTINUE | SKIP_FILE]

Export

COPY INTO 'target_path'
FROM { table_name | (SELECT ...) }
FILE_FORMAT = (TYPE = CSV | JSON | PARQUET [, option = value, ...])
[OVERWRITE = TRUE | FALSE]
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the existing COPY INTO SQL entry point and review how export currently accepts a table name as its source. Implement support for the proposed FROM (SELECT ...) form, then verify that arbitrary query results can be exported while preserving the existing format and overwrite options.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.