apache / apache/hudi

[SUPPORT] why is the schema evolution done while not setting hoodie.schema.on.read.enable

Open
#8,018 16 comments 0 reactions 1 assignee Claimed by @alexeykudinkin View on GitHub
area:schema area:writer
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

we have a glue streaming job that writes to hudi table, we try to do schema evolution, when we add a new col to any record, it works fine and the new col is shown when querying the table, the thing is we expect that it should not evolute the schema because we didn't set the config hoodie.schema.on.read.enable, and as we understand that this config is set by default to false, and as per hudi docs:

"Enables support for Schema Evolution feature
Default Value: false (Optional)
Config Param: SCHEMA_EVOLUTION_ENABLE"

so when didn't define it on our config, it shouldn't allow for the schema evolution and adding of the new columns, right?
we even tried to explicitly set it to false in our connection options, but still , when we add a new col it's shown to our table
**To Reproduce**

Steps to reproduce the behavior:

1. run the glue streaming job
2. add a record with new col/ attribute( attribute in case of dynamodb)
3. query the hudi table

**Expected behavior**

it shouldn't show the added cols/attributes as we disabled schema evolution and the col/attribute shouldn't be existing also in the schema of the table in the datalake.

**Environment Description**

* Hudi version : .12

* Spark version : 3

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : no
* glue version: 4

**Additional context**

our connection options are:
```
hudiWriteConfig = {
'className': 'org.apache.hudi',
'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
'hoodie.table.name': hudi_table_name,
'hoodie.datasource.write.table.name': hudi_table_name,
'hoodie.datasource.write.precombine.field': 'timestamp',
'hoodie.datasource.write.recordkey.field': 'user_id',
'hoodie.datasource.write.operation': 'upsert',
#"hoodie.compact.schedule.inline":"true",
'hoodie.datasource.hive_sync.use_jdbc':'false',
'hoodie.datasource.hive_sync.mode':'hms',
"hoodie.compact.inline": "true",
"hoodie.compact.inline.max.delta.commits":"3",
"hoodie.schema.on.read.enable":"false",
"hoodie.deltastreamer.schemaprovider.source.schema.file":"s3://hudi-test-table/menna/src.acsv",
"hoodie.deltastreamer.schemaprovider.target.schema.file":"s3://hudi-test-table/menna/target.acsv"
#'hoodie.datasource.write.partitionpath.field': 'year:SIMPLE,month:SIMPLE,day:SIMPLE',
#'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.CustomKeyGenerator',
#'hoodie.deltastreamer.keygen.timebased.timestamp.type': 'DATE_STRING',
#'hoodie.deltastreamer.keygen.timebased.input.dateformat': 'yyyy-mm-dd',
#'hoodie.deltastreamer.keygen.timebased.output.dateformat': 'yyyy/MM/dd'
}

hudiGlueConfig = {
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.sync_as_datasource': 'true',
'hoodie.datasource.hive_sync.database': database_name,
'hoodie.datasource.hive_sync.table': hudi_table_name,
'hoodie.datasource.hive_sync.use_jdbc': 'false',
'hoodie.datasource.write.hive_style_partitioning': 'false',
'hoodie.datasource.hive_sync.partition_extractor_class': 'org.apache.hudi.hive.MultiPartKeysValueExtractor',
#'hoodie.datasource.hive_sync.partition_fields': 'year,month,day'
}

commonConfig = {
'path': s3_path_hudi_table
}

combinedConf = {
**commonConfig,
**hudiWriteConfig,
**hudiGlueConfig
}
```
in glue streaming job we use:
```
glueContext.forEachBatch(
frame=data_frame_DataSource0,
batch_function=processBatch,
options={
"windowSize": window_size,
"checkpointLocation": s3_path_spark_checkpoints
}
)
```

and:
```
data_frame_DataSource0 = glueContext.create_data_frame.from_catalog(
database=database_name,
table_name=kinesis_table_name,
transformation_ctx="DataSource0",
additional_options={"inferSchema": "true", "startingPosition": starting_position_of_kinesis_iterator}
)
```
and the way we write our hudi table is:
```
kinesis_data_frame.write.format("hudi").options(**combinedConf).mode("append").save()
```
sometimes we write it as follows but it gives the same behaviour:
```
glueContext.write_dynamic_frame.from_options(
frame=DynamicFrame.fromDF(kinesis_data_frame, glueContext, "evolved_kinesis_data_frame"),
connection_type="custom.spark",
connection_options=combinedConf
)
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.