alibaba / alibaba/DataX

postgresqlreader Add support splitKey for timestamp column

Open
#201 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
17.4k
Forks
5.7k
PR merge metrics
No merged PRs in 30d

Description

In our use case we use alot of Timestamp column with indicate the created_at, updated_at.
Those fields usually was indexed.
Currently DataX was not support splitKey on non-numeric column.
To load big tables ~100 million records we only use 1 thread which is taking long time to load than requirement for hourly load.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the PostgreSQL reader and the existing splitKey handling for numeric columns. Trace how partition ranges and queries are built, then verify the design against indexed timestamp columns and large-table parallel loads; done means timestamp splitKey values can divide the read across multiple threads without breaking existing behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.