Additional Metadata Details for Hudi Transactions

Whenever following scenarios happen :
# Custom Datasource ( Kafka for instance ) -> Hudi Table
# Hudi -> Hudi Table
# s3 -> Hudi Table

Following metadata need to be captured :
# Table Level Metadata

*
** Operation name ( record level ) like Upsert, Insert etc for last operation performed on the row

# Transaction Level Metadata ( This will be logged on Hudi Level and not Table Level )
** Source ( Kafka Topic Name / S3 url for source data in case of s3 etc )
** Target Hudi Table Name
** Last transaction time ( last commit time )

Basically , point (1) collects all details on table level  and point (2) collects all the transactions happened on Hudi Level

Point(1) would be just a column addition for operation type

Eg for Point (2) :  Suppose we had an ingestion from Kafka topic 'A' to Hudi table 'ingest_kafka' and another ingestion from RDBMS table ( 'tableA' ) through Sqoop to Hudi Table 'RDBMSingest' then the metadata captured would be :

 
|Source|Timestamp|Transaction Type|Target|
|Kafka - 'A'|XXXXXX|UPSERT|ingest_kafka|
|RDBMS - 'tableA'|XXXXXX|INSERT|RDBMSingest|

 

The Transaction Details Table in Point (2) should be available as a separate common table which can be queried as Hudi Table or stored as parquet which can be queried from Spark

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1267
- Type: Improvement
- Affects version(s):
- 0.9.0

---

## Comments

03/Sep/20 22:37;vinoth;[~ashishmg] I originally thought that you want to query a single Hudi table around the timeline events, like what the CLI exposes today. 

 

But seems like you are looking at something across tables as well?;;;

---

04/Sep/20 02:09;ashishmg;[~vinoth] Yes, that would be good to have present timeline CLI metadata as a Table in Hudi. Maybe I can raise another Jira for the same if thats possible for present releases. The idea for this Jira adds more value in terms of audit purposes ( Point 2) which gives a view for the user to see all the transactions happening on the Data Lake . But if we can have Point 1 implemented in immediate releases, that would be great as Its a column addition of last operation performed on a row;;;

---

09/Sep/20 17:53;vinoth;ah got it. there was a proposal for a UI on top that reads across tables. this is worth discussing again on the mailing list.

 

This was the rough approach.
# We run a long running instance of  TimelineServer and have all the writers to each table report commits/have the server pull and materialize the table metadata in local rocksDB
# We can then build REST Layer on top of it and hook up a UI.

 ;;;

Start by reviewing Hudi's existing timeline CLI metadata and the issue's proposed table-level and cross-table transaction metadata. The comments point toward a TimelineServer and materialized metadata store, so clarify the design and scope before locating implementation entry points. Done means agreeing on the metadata model and making the resulting transaction details queryable as a Hudi table or Parquet.

Written by the indexing model from the issue text.