apache / apache/beam

Python SDK not creating Flink checkpoints

Open
#21,318 2 comments 0 reactions 0 assignees View on GitHub
bug core P3 python
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

I have been trying to get checkpointing to work with Apache Flink and Beam for the Python SDK, but without any success. I read a ton of documentation on how to get this working, but couldn't make any progress, so I have to assume that this is a bug. If not, then we need to at least fix the documentation (fundamentally).

The "bug" is reproducible with both the PortableRunner and the FlinkRunner **** uber JAR. I cannot really test the FlinkRunner without uber JARs, because I am submitting to a remote cluster.

The flink cluster is configured with:
```

   state.checkpoint-storage: "filesystem"
   state.checkpoints.dir: "file:///foo/bar/cp"
   state.savepoints.dir:
"file:///foo/bar/sp"
   execution.checkpointing.interval: "60s"
   execution.checkpointing.externalized-checkpoint-retention:
"DELETE_ON_CANCELLATION"
```

(`{}/foo/bar{`} is a shared network mount)

When I submit a job, all I'm seeing in the Flink Web UI under job configuration is
```

Execution mode: PIPELINED
Max. number of execution retries: Cluster level default restart strategy
Job
parallelism: 120
Object reuse mode: false
```

"User Configuration" is empty and no checkpoints are created (both the Checkpoints tab and the checkpoints folder remain empty).

I tried setting
```

checkpointing_interval=30000,
externalized_checkpoints_enabled=True,
```

in my Beam submission config, but the result is the same.

When I try the FlinkRunner with `{}flink_submit_uber_jar=True{`}, it's the same again, but this time I also get the following warning and the job starts with a parallelism of 1 (I guess that's another bug):
```

WARNING:apache_beam.options.pipeline_options:Discarding invalid overrides: {'checkpointing_interval':
30000, 'externalized_
checkpoints_enabled': True, 'parallelism': 120}
```

Imported from Jira [BEAM-13675](https://issues.apache.org/jira/browse/BEAM-13675). Original Jira may contain additional context.
Reported by: phoerious.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.