lightning: column-constants config for injecting literal values into missing/ignored columns
- Dominant language
- Go
- Stars
- 40.5k
- Forks
- 6.2k
- PR merge metrics
- PR metrics pending
Description
## Feature Request
### Problem Summary
In ETL pipelines that ingest into a subset of a partitioned table's columns, certain columns (e.g. \`customer_name\`, \`etl_ts\`) are not present in the source CSV but must be populated with known constant values per-import job. Currently there is no way to configure this without pre-processing the source files or creating custom views.
### Proposed Solution
Add a \`[[mydumper.column-constants]]\` section that injects a literal string value for any column absent from the source data file. Each entry targets a specific table (by \`db\`/\`table\` or \`table-filter\`) and provides a \`values\` map of column name → constant string:
```toml
[[mydumper.column-constants]]
db = "mydb"
table = "mytable"
[mydumper.column-constants.values]
customer_name = "acme"
etl_ts = "2026-04-17 21:00:00"
```
Lightning casts the string constant to the column's target type using the same \`CastColumnValue\` path as other column encoding. If the cast fails (e.g. \`"not_a_number"\` into a \`BIGINT\`), Lightning returns an error at encode time.
**Note:** Columns listed in \`column-constants\` are automatically added to the ignore-columns set — you do not need to list them separately under \`[[mydumper.ignore-data-columns]]\`.
### Behaviour
- Constants apply when the column is absent from the source row (\`inputDatum == nil\`)
- Constants also apply when the source explicitly sends \`NULL\` for a \`NOT NULL\` column (overrides \`HandleBadNull\`)
- Constants do **not** override an explicit non-NULL value from the source
- Keys in \`values\` are normalized to lowercase at config load time (case-insensitive)
- The schema pre-check treats a column with a configured constant as satisfying the "has a default value" requirement
- Each entry supports either \`db\`/\`table\` (exact match) or \`table-filter\` (glob pattern) — not both
### Primary Key Behaviour
Constants **cannot** override `AUTO_INCREMENT` or `AUTO_RANDOM` primary keys. The encoder handles these columns before it reaches the constant lookup — they always use the Lightning-assigned row ID regardless of any configured constant.
For a plain primary key (no `AUTO_INCREMENT`/`AUTO_RANDOM`) that is absent from the source data, a constant would technically be applied. However this is not a supported use case — all rows would receive the same PK value, causing a unique constraint violation on import.
### Use Case
TiDB Lightning ETL pipelines doing partition-targeted ingest where each job corresponds to a known customer or time-slice value.
Issue Number: N/A
Contributor guide
Assessment
This issue has not been assessed yet.