[Bug]: Execution of pipeline fails when CSV File Input transform has Lazy Conversion flag ON
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 33m
- Merged PRs (30d)
- 204
Description
### Apache Hop version?
2.7.0 (2023-11-17 12.19.09)
### Java version?
openjdk version "11.0.22" 2024-01-16 LTS
### Operating system
macOS
### What happened?
When CSV File Input transforms used in this attached pipeline ((rename to .hpl); with input csv files) is set ON, execution of this pipeline fails with following errors while tried on Spark engine -
2024/03/16 15:07:00 - aggr1 - ERROR: java.lang.RuntimeException: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder)))
2024/03/16 15:07:00 - aggr1 - Caused by: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder)))2024/03/16 15:07:00 - aggr1 - Caused by: java.io.EOFException
2024/03/16 15:07:00 - aggr1 - at java.base/java.io.ObjectInputStream$BlockDataInputStream.readBoolean(ObjectInputStream.java:3370)
Entire error stack trace attached.
Same pipeline with Lazy Conversion flags turn off in all 3 CSV File Input transforms, pipeline runs fine on Spark -
2024/03/16 15:11:17 - aggr1 - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/saikatbasu/hop/hopClient/plugins/engines/beam/lib/spark-unsafe_2.12-3.4.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024/03/16 15:11:45 - aggr1 - Beam pipeline execution has finished.
[SparkExecFineWithLzConvOff.txt](https://github.com/apache/hop/files/14625428/SparkExecFineWithLzConvOff.txt)
[SparkExecErrorWithLzConvOn.txt](https://github.com/apache/hop/files/14625429/SparkExecErrorWithLzConvOn.txt)
[aggr1.hpl.txt](https://github.com/apache/hop/files/14625431/aggr1.hpl.txt)
[SSB.DATES.csv](https://github.com/apache/hop/files/14625432/SSB.DATES.csv)
[SSB.LINEORDER.csv](https://github.com/apache/hop/files/14625433/SSB.LINEORDER.csv)
[SSB.CUSTOMER.csv](https://github.com/apache/hop/files/14625434/SSB.CUSTOMER.csv)
### Issue Priority
Priority: 3
### Issue Component
Component: Beam, Component: Hop Gui, Component: Pipelines
Contributor guide
Research direction
Reproduce the failure using the attached aggr1.hpl pipeline and CSV inputs with Lazy Conversion enabled on the Beam Pipeline Engine with the Spark run configuration. Compare the failing and successful logs, SparkExecErrorWithLzConvOn.txt and SparkExecFineWithLzConvOff.txt, and trace the CSV File Input transforms through the Beam execution path. Done means the pipeline runs successfully with Lazy Conversion enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100