postgresqlreader Add support splitKey for timestamp column
- Dominant language
- Java
- Stars
- 17.4k
- Forks
- 5.7k
- PR merge metrics
- No merged PRs in 30d
Description
In our use case we use alot of Timestamp column with indicate the created_at, updated_at.
Those fields usually was indexed.
Currently DataX was not support splitKey on non-numeric column.
To load big tables ~100 million records we only use 1 thread which is taking long time to load than requirement for hourly load.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the PostgreSQL reader and the existing splitKey handling for numeric columns. Trace how partition ranges and queries are built, then verify the design against indexed timestamp columns and large-table parallel loads; done means timestamp splitKey values can divide the read across multiple threads without breaking existing behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, postgresql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100