Incubating Program: make PITR production-ready

Introduce

github: https://github.com/lvleiice/Better-PITR

PITR is an ecosystem tool for TiDB Binlog. By preprocessing the incremental backup file of TiDB, PITR merged the changes of the same line of data to produce a new, lighter incremental backup file, which greatly reduced the Time of incremental backup Recovery and realized fast-pitr (Fast Point in Time Recovery).

For example

There is a table t1, it's schema is: create table t1 (id int primary key, name varchar(24)). And now we execute four SQLs in TiDB:

insert into t1 values(1, “a”);
insert into t1 values(2, “b”);
update t1 set name = “c” where id = 1;
delete from t1 where id = 2;

These SQLs will generate four binlog, restore binlog using Reparo tool data to downstream will execute four SQLs in downstream database. These binlogs are actually can merged to generate an insert into t1 values(1, "c"); This will save a quarter as much space as before and restore the files four times as fast. We can think of it simply: the binlog file produced by Drainer is compressed/preprocessed by PITR.

Current Situation

PITR is a Hackathon project, so it only implements the basic functionality, has some known problems, and lacks testing, so there may be more unknown problems. We need to solve the below problems, and make PITR production-ready.

Bug

• PITR needs to maintain the table structure information (to parse the binlog data) but will report an error if there is no database information in the DDL.

Performance

• PITR retrieves TiDB's historical DDL information to build the table structure of the initial state, and executes the DDL in PITR's built-in TiDB to retrieve the table structure. If the history of DDL is large, the initialization state can be very slow. (defective design)

Test

• Unit test coverage was 63.8%
• No integration testing

Usability

• PITR does not output meta information, such as the location of the file to run the processing, or the time period during which binlog was processed (the binlog commit ts at the beginning and the binlog commit ts at the end).

Estimated Time

3(Developers) * 7(days)

Start with the Better-PITR project linked in the issue and review its current implementation against the listed bug, initialization-performance problem, testing gaps, and missing metadata. Done means resolving the database-information failure, improving slow historical-DDL initialization, adding unit and integration coverage, and reporting the processed file location and commit-timestamp range.