apache / apache/hudi

Test coverage for table creation

Open
#16,808 1 comment 0 reactions 1 assignee Assigned to @yihua View on GitHub
area:tests from-jira priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

[follow up of
https://github.com/apache/hudi/pull/12615|https://github.com/apache/hudi/pull/12615]
goal for this patch (open to splitting it up into multiple):
* for new table creation, we should default to table version 8.
* for new table creation, if user overrides "hoodie.write.table.version", we will honor the overridden value.

We need to validate and may be fix all writers (spark-ds, spark-sql, deltastreamer and spark structured streaming ingestion)

 
|Deltastreamer :
[hudi/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java|https://github.com/apache/hudi/blob/ef5b9ca3777775fae29c04c06cbb743183e9eef3/hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java#L421]
Line 421 in [ef5b9ca|https://github.com/apache/hudi/commit/ef5b9ca3777775fae29c04c06cbb743183e9eef3]
||HoodieTableMetaClient initializeEmptyTable(HoodieTableMetaClient.TableBuilder tableBuilder, String partitionColumns,|

[hudi/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java|https://github.com/apache/hudi/blob/ef5b9ca3777775fae29c04c06cbb743183e9eef3/hudi-utilities/src/test/java/org/apache/hudi/utilities/deltastreamer/TestHoodieDeltaStreamer.java#L428]
Line 428 in [ef5b9ca|https://github.com/apache/hudi/commit/ef5b9ca3777775fae29c04c06cbb743183e9eef3]
||public void testTableCreation() throws Exception {|
 
spark-sql and spark-ds: guess this patch already have tests.
spark-streaming:
[hudi/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala|https://github.com/apache/hudi/blob/ef5b9ca3777775fae29c04c06cbb743183e9eef3/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/TestStructuredStreaming.scala#L208]
Line 208 in [ef5b9ca|https://github.com/apache/hudi/commit/ef5b9ca3777775fae29c04c06cbb743183e9eef3]
||def testStructuredStreaming(tableType: HoodieTableType): Unit = {|
 |
 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8914
- Type: Test
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.