unitycatalog / unitycatalog/unitycatalog

unitycatalog-hadoop follow-up works

Open
#1,572 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
3.5k
Forks
672
Avg merge
6d 16h
Merged PRs (30d)
30

Description

Comment 1: in this comment: https://github.com/unitycatalog/unitycatalog/pull/1569#discussion_r3307146324, @yili-db suggested us to refactor the CredPropsUtil, so that we can reuse most of the code for all the createXCredProps.

Here is the original comment:

It looks like the framework of all 4 createXCredProps functions are the same, except that each version has a customized way to specify the "identity" when renewCredEnabled==true:

In this function it's .deltaStagingTableId(stagingTableId, location) for all three clouds
In createTableCredProps it's .credentialType(UCHadoopConfConstants.UC_CREDENTIALS_TYPE_TABLE_VALUE).tableId(tableId).tableOperation(tableOp) etc.
So it appears that they can be made a "Base" function that does this framework for all 3 clouds, then each createXCredProps function provide a lambda that sets their customized keys when renewCredEnabled==true?

Comment 2: In this oss-delta's UCDeltaTokenBasedRestClient, we will need a PR to remove the TODO, and file a separate PR to use the latest buildForStagingTable: https://github.com/delta-io/delta/blob/4982fbc8d9d8c7d20106716b2ec8d5a61d53618d/storage/src/main/java/io/delta/storage/commit/uccommitcoordinator/UCDeltaTokenBasedRestClient.java#L215-L227

@yili-db pls help us to file a separate PR to address this in the OSS delta project.

Comment 3: In this UCHadoopConfConstants, we will need to carefully review those hadoop properties: https://github.com/unitycatalog/unitycatalog/blob/bdfbef9af32be7a9d8ab6ccbc93ce8cf83908d0d/connectors/hadoop/src/main/java/io/unitycatalog/hadoop/internal/UCHadoopConfConstants.java#L7

and see what are the keys that used for legacy UC API and what are the for UC Delta API.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with CredPropsUtil and the four createXCredProps functions described in the issue, then review connectors/hadoop/src/main/java/io/unitycatalog/hadoop/internal/UCHadoopConfConstants.java. Separately inspect oss-delta's UCDeltaTokenBasedRestClient around buildForStagingTable and the TODO referenced in the issue. Done should include a shared credential-properties structure, the requested Delta follow-up, and a clear separation of legacy UC and UC Delta Hadoop properties.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.