[Bug]: NullPointerException Issue with Executing generate-synthetic-data.hpl Pipeline in Apache Flink
- Dominant language
- Java
- Stars
- 1.5k
- Forks
- 476
- Avg merge
- 18h 32m
- Merged PRs (30d)
- 216
Description
### Apache Hop version?
2.12
### Java version?
17
### Operating system
Windows
### What happened?
Hi team,
I am encountering an issue while executing the generate-synthetic-data.hpl sample pipeline using Apache Flink. Here’s what I’ve configured:
- Set the Engine Type to Beam Flink pipeline engine
- Configured the Flink master
- Mentioned the hop-fat-jar.jar
However, when running the pipeline, I am facing the exception mentioned below. I am using Apache Hop version 2.12 on Windows and Apache Flink version 1.17.0, which is running on Linux. From the Flink UI, I can see that the job is going into a failed state.
> org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster.
> at org.apache.flink.runtime.jobmaster.DefaultJobMasterServiceProcess.lambda$new$0(DefaultJobMasterServiceProcess.java:97)
> at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1595)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.CompletionException: java.lang.NullPointerException
> at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273)
> at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at org.apache.flink.api.common.ExecutionConfig.getNumberOfExecutionRetries(ExecutionConfig.java:486)
> at org.apache.flink.api.common.ExecutionConfig.getRestartStrategy(ExecutionConfig.java:459)
> at org.apache.flink.runtime.scheduler.DefaultSchedulerFactory.createInstance(DefaultSchedulerFactory.java:99)
> at org.apache.flink.runtime.jobmaster.DefaultSlotPoolServiceSchedulerFactory.createScheduler(DefaultSlotPoolServiceSchedulerFactory.java:119)
> at org.apache.flink.runtime.jobmaster.JobMaster.createScheduler(JobMaster.java:371)
> at org.apache.flink.runtime.jobmaster.JobMaster.(JobMaster.java:348)
> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.internalCreateJobMasterService(DefaultJobMasterServiceFactory.java:123)
> at org.apache.flink.runtime.jobmaster.factories.DefaultJobMasterServiceFactory.lambda$createJobMasterService$0(DefaultJobMasterServiceFactory.java:95)
> at org.apache.flink.util.function.FunctionUtils.lambda$uncheckedSupplier$4(FunctionUtils.java:112)
> at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
Has anyone encountered a similar issue or could provide some insights on how to resolve this?
Thanks in advance for your help!
### Issue Priority
Priority: 2
### Issue Component
Component: Pipelines
Contributor guide
Research direction
Start by reproducing the generate-synthetic-data.hpl sample pipeline with Apache Hop 2.12, the Beam Flink engine, Flink 1.17.0, and the configured hop-fat-jar.jar. Inspect the Flink job setup around the reported ExecutionConfig.getNumberOfExecutionRetries stack trace. Done means the sample pipeline starts successfully on the configured Linux Flink cluster instead of entering a failed state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100