unitycatalog / unitycatalog/unitycatalog
unitycatalog-hadoop follow-up works
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3.5k
- Forks
- 672
- Avg merge
- 6d 16h
- Merged PRs (30d)
- 30
Description
Comment 1: in this comment: https://github.com/unitycatalog/unitycatalog/pull/1569#discussion_r3307146324, @yili-db suggested us to refactor the CredPropsUtil, so that we can reuse most of the code for all the createXCredProps.
Here is the original comment:
It looks like the framework of all 4 createXCredProps functions are the same, except that each version has a customized way to specify the "identity" when renewCredEnabled==true:
In this function it's .deltaStagingTableId(stagingTableId, location) for all three clouds
In createTableCredProps it's .credentialType(UCHadoopConfConstants.UC_CREDENTIALS_TYPE_TABLE_VALUE).tableId(tableId).tableOperation(tableOp) etc.
So it appears that they can be made a "Base" function that does this framework for all 3 clouds, then each createXCredProps function provide a lambda that sets their customized keys when renewCredEnabled==true?
Comment 2: In this oss-delta's UCDeltaTokenBasedRestClient, we will need a PR to remove the TODO, and file a separate PR to use the latest buildForStagingTable: https://github.com/delta-io/delta/blob/4982fbc8d9d8c7d20106716b2ec8d5a61d53618d/storage/src/main/java/io/delta/storage/commit/uccommitcoordinator/UCDeltaTokenBasedRestClient.java#L215-L227
@yili-db pls help us to file a separate PR to address this in the OSS delta project.
Comment 3: In this UCHadoopConfConstants, we will need to carefully review those hadoop properties: https://github.com/unitycatalog/unitycatalog/blob/bdfbef9af32be7a9d8ab6ccbc93ce8cf83908d0d/connectors/hadoop/src/main/java/io/unitycatalog/hadoop/internal/UCHadoopConfConstants.java#L7
and see what are the keys that used for legacy UC API and what are the for UC Delta API.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with CredPropsUtil and the four createXCredProps functions described in the issue, then review connectors/hadoop/src/main/java/io/unitycatalog/hadoop/internal/UCHadoopConfConstants.java. Separately inspect oss-delta's UCDeltaTokenBasedRestClient around buildForStagingTable and the TODO referenced in the issue. Done should include a shared credential-properties structure, the requested Delta follow-up, and a clear separation of legacy UC and UC Delta Hadoop properties.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- backend, data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100