pingcap / pingcap/tidb

Add new option `remove_csv_header` for `IMPORT INTO`

Open
#61,166 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Enhancement

Introduce a new option for CSV Import: `remove_csv_header`, which is intended to replace `skip_rows` option. `remove_csv_header` has three values:
- **`true`**: the first row of each CSV file will always be skipped, which is equivalent to `skip_rows=1`
- **`false`**: all rows will be treated as data, which is equivalent to `skip_rows=1`
- **`auto`**: read the first row of each CSV, and compare with the column names of the imported table. If all the fields match, the first row will be skipped.

To be mentioned, `auto` is incompatible with `ColumnNameOrUserVarList`, you can only use `auto` for the following SQL:
```SQL
IMPORT INTO db.table FROM "XXX" WITH REMOVE_CSV_HEADER="AUTO";
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the IMPORT INTO implementation and its handling of the existing skip_rows option, then inspect related CSV import tests. Define coverage for remove_csv_header=true, false, and auto, including the incompatibility with ColumnNameOrUserVarList; done means the documented SQL forms behave as specified and tests cover each mode.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.