pingcap / pingcap/tidb

lightning: column-constants config for injecting literal values into missing/ignored columns

Open
#67,896 0 comments 0 reactions 0 assignees View on GitHub
contribution
Dominant language
Go
Stars
40.5k
Forks
6.2k
PR merge metrics
PR metrics pending

Description

## Feature Request

### Problem Summary

In ETL pipelines that ingest into a subset of a partitioned table's columns, certain columns (e.g. \`customer_name\`, \`etl_ts\`) are not present in the source CSV but must be populated with known constant values per-import job. Currently there is no way to configure this without pre-processing the source files or creating custom views.

### Proposed Solution

Add a \`[[mydumper.column-constants]]\` section that injects a literal string value for any column absent from the source data file. Each entry targets a specific table (by \`db\`/\`table\` or \`table-filter\`) and provides a \`values\` map of column name → constant string:

```toml
[[mydumper.column-constants]]
db = "mydb"
table = "mytable"
[mydumper.column-constants.values]
customer_name = "acme"
etl_ts = "2026-04-17 21:00:00"
```

Lightning casts the string constant to the column's target type using the same \`CastColumnValue\` path as other column encoding. If the cast fails (e.g. \`"not_a_number"\` into a \`BIGINT\`), Lightning returns an error at encode time.

**Note:** Columns listed in \`column-constants\` are automatically added to the ignore-columns set — you do not need to list them separately under \`[[mydumper.ignore-data-columns]]\`.

### Behaviour

- Constants apply when the column is absent from the source row (\`inputDatum == nil\`)
- Constants also apply when the source explicitly sends \`NULL\` for a \`NOT NULL\` column (overrides \`HandleBadNull\`)
- Constants do **not** override an explicit non-NULL value from the source
- Keys in \`values\` are normalized to lowercase at config load time (case-insensitive)
- The schema pre-check treats a column with a configured constant as satisfying the "has a default value" requirement
- Each entry supports either \`db\`/\`table\` (exact match) or \`table-filter\` (glob pattern) — not both

### Primary Key Behaviour

Constants **cannot** override `AUTO_INCREMENT` or `AUTO_RANDOM` primary keys. The encoder handles these columns before it reaches the constant lookup — they always use the Lightning-assigned row ID regardless of any configured constant.

For a plain primary key (no `AUTO_INCREMENT`/`AUTO_RANDOM`) that is absent from the source data, a constant would technically be applied. However this is not a supported use case — all rows would receive the same PK value, causing a unique constraint violation on import.

### Use Case

TiDB Lightning ETL pipelines doing partition-targeted ingest where each job corresponds to a known customer or time-slice value.

Issue Number: N/A

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.