apache / apache/streampark

[Bug] Build result paths are silently dropped when persisted, breaking every subsequent job start

Open
#4,488 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
4.3k
Forks
1.1k
Avg merge
13h 35m
Merged PRs (30d)
2

Description

### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/streampark/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues referencing `ShadedBuildResponse` / `buildResultJson`.

### Java Version

Temurin 21.0.11 (console), built with Microsoft OpenJDK 11.0.28

### Scala Version

2.12.x

### StreamPark Version

3.0.0-SNAPSHOT (`dev` branch, commit `9ddda84c9`)

### Flink Version

1.20.4 and 2.2.1 (official binary distributions, standalone/remote cluster)

### Deploy mode

remote

### What happened

A Flink SQL application builds successfully — `POST /flink/pipe/detail` reports `pipeStatus=3`, both steps `success`, and the fat jar really is on disk with the correct `Main-Class` — but the following `start` fails. Depending on which downstream path consumes the build result, the failure surfaces as one of three unrelated-looking errors:

- `NullPointerException` from `new File(null)` in `SubmitRequest.userJarFile()`
- `The program's entry point class 'org.apache.streampark.flink.cli.SqlClient' was not found in the jar file`
- `NullPointerException: Cannot invoke "java.io.File.getName()" because "flinkJobJar" is null`

The giveaway is in `logs/streampark.out`:

```
flinkBuildResult : { workspacePath: null, shadedJarPath: null, pass: true }
```

The build succeeded, but the object read back from the database has lost its paths.

**Root cause**: the getters on `AbstractFlinkBuildResponse`, `ShadedBuildResponse`, `K8sAppModeBuildResponse` and `DockerImageBuildResponse` do not follow JavaBean naming (`shadedJarPath()`, `workspacePath()`, `pass()`, `flinkBaseImage()`, `mainJarPath()`, `extraLibJarPaths()`, `flinkImageTag()`, `podTemplatePaths()`, `dockerInnerMainJarPath()` — no `get` prefix) and carry no `@JsonProperty`. `JacksonUtils` uses a default `ObjectMapper`, whose bean introspection does not recognise them as getters, so it **silently omits** those fields. `ApplicationBuildPipeline.onFinish()` therefore writes an incomplete JSON into `t_flink_app`'s `buildResultJson`, and `start()` deserialises that incomplete JSON back.

`pass` is the reason this went unnoticed for so long: it is dropped on write too, but its field default is already `true`, so it reads back looking correct.

This also affects the K8s Application and Docker image build responses, whose image and jar paths are persisted the same way.

Reproduced by round-tripping the response through a plain `ObjectMapper` in `jshell` against the built jar: `shadedJarPath` is absent from the serialised JSON before the fix and present after.

### Error Exception

```log
java.lang.NullPointerException: Cannot invoke "java.io.File.getName()" because "flinkJobJar" is null
at org.apache.streampark.flink.client.impl.FlinkSessionSubmitHelper.doSubmitViaRestApi(...)
```

### Are you willing to submit PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with AbstractFlinkBuildResponse, ShadedBuildResponse, K8sAppModeBuildResponse, DockerImageBuildResponse, JacksonUtils, and ApplicationBuildPipeline.onFinish(). Reproduce the plain ObjectMapper round trip described in the issue, then verify that build paths and related fields survive persistence and that subsequent start no longer receives null paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, java, kubernetes
Domain
backend, database
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.