DynamoDB based external index implementation

We have a HBaseIndex, that provides uses with ability to store fileID <=> recordKey mappings in an external kv store, for fast lookups during upsert operations. We can potentially create a similar one for DynamoDB.

We just use a single column family in HBase, so we should be able to largely re-use the code/key-value schema across them even.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-2199
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-3966

---

## Comments

01/Aug/21 09:53;Biswajit11;so we will need properties for dynamodb also , like in emr we have a table name emrfs , can we have a table like that called as hudifs which gets created if the table doesn't exist ?

 Just one question does it creates a table for each table name ingested or it uses a generic table for table ingested into hudi ?

 

also can you say me the class name for hbase from where i can get an idea ?

 

 

EDIT 

 

i was just checking some hbase implementation classes in hudi 

 

i found out it stores the data in this format 

partition_path  -> is it string ?

, fileID  ---> string or number ?

 

commitTime   -> new Date().getTime();

 

 

 

As in dynamodb there is a concept of PK and SK can you let me know what would be the pk and sk for this would be and can you give an example of data that gets stored here ?

 

 ;;;

---

04/Aug/21 05:19;vinoth;> it creates a table for each table name ingested or it uses a generic table for table ingested into hudi ?

IMO, having separate tables is better for manageability. 

 

[https://github.com/apache/hudi/blob/master/hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/hbase/SparkHoodieHBaseIndex.java] 

here you go!;;;

---

03/Oct/21 06:36;Biswajit11;Thanks , was on a vacation will start working on this ;;;

---

24/Dec/21 01:43;Biswajit11;Insert and upsert is completed and testing is in progress , rollback is still left 

As dynamodb only can efficiently search using hash key and sort key and both of them are already used by hudi primary key and partition key so using commit_ts one gsi index needs to be created for efficent rollback ;;;

---

20/Sep/22 19:21;tgraciadio;Hi
Do we have any updates on this feature?;;;

Start with the HBaseIndex implementation, especially hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/index/hbase/SparkHoodieHBaseIndex.java, and compare its external key-value schema and lookup flow. Define the DynamoDB table and key design for upsert and rollback, then verify that the implementation and its tests cover both operations.

Written by the indexing model from the issue text.