apache / apache/hudi

[SUPPORT] AWSGlueCatalogSyncClient.updateTableComments applies no comments since AWS SDK v2 upgrade

Open
#19,316 0 comments 0 reactions 1 assignee Claimed by @rangareddy View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

`AWSGlueCatalogSyncClient.updateTableComments` no longer applies any column or partition column comments since the AWS SDK v2 upgrade.

Its helper `setComments` builds a new `Column` and discards the result instead of replacing the element in the list:

```java
private void setComments(List columns, Map> commentsMap) {
columns.forEach(column -> {
String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
Column.builder().comment(comment).build(); // result dropped, column is unchanged
});
}
```

AWS SDK v2 model classes are immutable, so mutating in place is not possible. Before the SDK v2 upgrade (e9dd73fc08fd, #9347) this code called `column.setComment(...)` on the mutable v1 model, which worked. As a result `updateTableComments` never detects a change and always returns `false`, and no comments are synced to Glue on the update path.

The existing Glue comment tests do not catch this because they only assert comments that were pre-baked into the mocked `Column` objects.

**Expected behavior**

With `hoodie.datasource.hive_sync.sync_comment=true`, column and partition column comments should be applied to the Glue table.

**Possible fix**

Rebuild the column lists with the comment applied, e.g.:

```java
for (int i = 0; i < columns.size(); i++) {
Column column = columns.get(i);
String comment = commentsMap.getOrDefault(column.name(), Option.empty()).orElse(null);
columns.set(i, column.toBuilder().comment(comment).build());
}
```

(using mutable copies of `storageDescriptor.columns()` / `table.partitionKeys()`, which are returned as unmodifiable lists by the SDK), and update the tests to assert comments that were not already present on the input columns.

**Environment Description**

* Hudi version: master (1.3.0-SNAPSHOT), broken since the AWS SDK v2 upgrade (#9347)

**Additional context**

Found during review of #19289, which fixes the equivalent HMS paths. Related to HUDI-8843 / #17359.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.