Flink 1.18 array values incorrect in fg reader test
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
{code:java}
<{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd", "partition_path": "any_partition", "trip_type": "BLACK", "rider": "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925, "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon": 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch": -9002173267385180332, "weight": 0.9950906, "nation": "Canada", "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97], "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701, "currency": "USD"}, "tip_history": [{"amount": 78.67847272460095, "currency": "USD"}], "customField0": 1299566198, "customField1": 5724776041695989483, "customField2": 0.7481167, "customField3": 0.2553663812411848, "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362", "customField5": "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted": false}> <{"timestamp": 0, "_row_key": "30f916be-06d6-4ef3-a9fb-9dddc40e8abd", "partition_path": "any_partition", "trip_type": "BLACK", "rider": "rider-001", "driver": "driver-001", "begin_lat": 0.7585664392201925, "begin_lon": 0.37789349980839115, "end_lat": 0.7757098794857531, "end_lon": 0.1730947142293684, "distance_in_meters": 1356260410, "seconds_since_epoch": -9002173267385180332, "weight": 0.9950906, "nation": "Canada", "current_date": 13, "current_ts": 1182257224, "height": [0, 0, 11, 121, -97], "city_to_state": {"LA": "CA"}, "fare": {"amount": 89.22144149794701, "currency": "USD"}, "tip_history": [{"amount": 4.5791476348245315, "currency": "USD"}], "customField0": 1299566198, "customField1": 5724776041695989483, "customField2": 0.7481167, "customField3": 0.2553663812411848, "customField4": "25db73af-fdf9-4e04-b47d-ce44882bf362", "customField5": "877c2613-f2a1-4afd-af4a-863814dec38b", "_hoodie_is_deleted": false}>
{code}
every field is the same except for the tip_history. This only happens on Flink 1.18 and not 1.20. I added a config to HoodieTestDataGenerator.SchemaEvolutionConfigs that can be enabled to expose this issue. I have marked it as a TODO in the code
I read the data in spark to ensure it wasn't an issue with the writer:
{code:java}
scala> spark.read.format("hudi").load("/var/folders/d0/l7mfhzl1661byhh3mbyg5fv00000gn/T/junit-5387917324981749197/").select("_row_key","tip_history").show(100,false)
25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Properties file file:/etc/hudi/conf/hudi-defaults.conf not found. Ignoring to load props file
25/07/18 14:12:07 WARN DFSPropertiesConfiguration: Cannot find HUDI_CONF_DIR, please set it as the dir of hudi-defaults.conf
# WARNING: Unable to attach Serviceability Agent. Unable to attach even with module exceptions: [org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed., org.apache.hudi.org.openjdk.jol.vm.sa.SASupportException: Sense failed.]
+------------------------------------+---------------------------+
|_row_key |tip_history |
+------------------------------------+---------------------------+
|30f916be-06d6-4ef3-a9fb-9dddc40e8abd|[{78.67847272460095, USD}] |
|d0efa8ad-5f8e-4ee2-ace7-963585aa2b7f|[{94.05111350973606, USD}] |
|f0dd358d-de15-4c67-9e3b-f006263d9286|[{30.4615021527871, USD}] |
|e6db310d-71ca-4147-9f4f-cbf7a2675016|[{3.853885568405202, USD}] |
|27819509-b6ac-45ee-8a7a-a031b2ef9a05|[{4.5791476348245315, USD}]|
+------------------------------------+---------------------------+ {code}
The tip history matches the expected 78.67847272460095 for record 30f916be-06d6-4ef3-a9fb-9dddc40e8abd.
Additionally, we can see that the value that is showing up in the record exactly matches the value in the last record `27819509-b6ac-45ee-8a7a-a031b2ef9a05`
I went back and looked at all the expected records and they all have that same value in their tip history
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9603
- Type: Bug
- Affects version(s):
- 1.0.2
- Fix version(s):
- 1.1.1
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.