apache / apache/hop

[Bug]: Execution of pipeline fails when CSV File Input transform has Lazy Conversion flag ON

Open
#3,724 0 comments 0 reactions 0 assignees View on GitHub
awaiting triage Beam Hop Gui P3 Pipelines
Dominant language
Java
Stars
1.5k
Forks
476
Avg merge
18h 33m
Merged PRs (30d)
204

Description

### Apache Hop version?

2.7.0 (2023-11-17 12.19.09)

### Java version?

openjdk version "11.0.22" 2024-01-16 LTS

### Operating system

macOS

### What happened?

When CSV File Input transforms used in this attached pipeline ((rename to .hpl); with input csv files) is set ON, execution of this pipeline fails with following errors while tried on Spark engine -

2024/03/16 15:07:00 - aggr1 - ERROR: java.lang.RuntimeException: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder)))
2024/03/16 15:07:00 - aggr1 - Caused by: java.lang.IllegalStateException: Error decoding bytes for coder: WindowedValue$ValueOnlyWindowedValueCoder(KvCoder(BigEndianLongCoder,PCollectionViews$ValueOrMetadataCoder(HopRowCoder,OffsetRange$Coder)))2024/03/16 15:07:00 - aggr1 - Caused by: java.io.EOFException
2024/03/16 15:07:00 - aggr1 - at java.base/java.io.ObjectInputStream$BlockDataInputStream.readBoolean(ObjectInputStream.java:3370)

Entire error stack trace attached.

Same pipeline with Lazy Conversion flags turn off in all 3 CSV File Input transforms, pipeline runs fine on Spark -

2024/03/16 15:11:17 - aggr1 - Executing this pipeline using the Beam Pipeline Engine with run configuration 'Spark'
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.spark.unsafe.Platform (file:/Users/saikatbasu/hop/hopClient/plugins/engines/beam/lib/spark-unsafe_2.12-3.4.0.jar) to constructor java.nio.DirectByteBuffer(long,int)
WARNING: Please consider reporting this to the maintainers of org.apache.spark.unsafe.Platform
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2024/03/16 15:11:45 - aggr1 - Beam pipeline execution has finished.


[SparkExecFineWithLzConvOff.txt](https://github.com/apache/hop/files/14625428/SparkExecFineWithLzConvOff.txt)
[SparkExecErrorWithLzConvOn.txt](https://github.com/apache/hop/files/14625429/SparkExecErrorWithLzConvOn.txt)
[aggr1.hpl.txt](https://github.com/apache/hop/files/14625431/aggr1.hpl.txt)
[SSB.DATES.csv](https://github.com/apache/hop/files/14625432/SSB.DATES.csv)
[SSB.LINEORDER.csv](https://github.com/apache/hop/files/14625433/SSB.LINEORDER.csv)
[SSB.CUSTOMER.csv](https://github.com/apache/hop/files/14625434/SSB.CUSTOMER.csv)

### Issue Priority

Priority: 3

### Issue Component

Component: Beam, Component: Hop Gui, Component: Pipelines

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure using the attached aggr1.hpl pipeline and CSV inputs with Lazy Conversion enabled on the Beam Pipeline Engine with the Spark run configuration. Compare the failing and successful logs, SparkExecErrorWithLzConvOn.txt and SparkExecFineWithLzConvOff.txt, and trace the CSV File Input transforms through the Beam execution path. Done means the pipeline runs successfully with Lazy Conversion enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.