lyft / lyft/flinkk8soperator

Incremental Checkpointing not work

Open
#203 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
581
Forks
153
PR merge metrics
No merged PRs in 30d

Description

i have a yaml in k8s like below:

> flinkConfig:
jobmanager.heap.size: "2g"
jobmanager.execution.failover-strategy: region
jobmanager.web.log.path: /var/log/jobmanager/current
taskmanager.log.path: /var/log/taskmanager/current
taskmanager.exit-on-fatal-akka-error: true
taskmanager.network.memory.min: 128m
taskmanager.network.memory.max: 512m
taskmanager.network.memory.fraction: 0.125
taskmanager.maxRegistrationDuration: Inf
taskmanager.initial-registration-pause: 500ms
taskmanager.max-registration-pause: 5s
taskmanager.refused-registration-pause: 5s
taskmanager.heap.size: "2g"
web.upload.dir: /opt/flink
akka.framesize: 20m
parallelism.default: 1
state.backend: rocksdb
state.backend.incremental: true
state.backend.fs.checkpointdir: hdfs://.../checkpoints-backend
state.checkpoints.dir: hdfs://.../checkpoints
state.savepoints.dir: hdfs://.../savepoints
restart-strategy: fixed-delay
restart-strategy.fixed-delay.delay: 10000s
restart-strategy.fixed-delay.attempts: 6
savepointPath: hdfs://*/savepoints

but i found that Incremental Checkpointing didn't work, what's wrong with it.thanks for your replay

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are identified. Start by checking the operator's handling of the shown flinkConfig and the RocksDB, HDFS checkpoint, and incremental-checkpoint settings, then reproduce the reported behavior with this Kubernetes configuration. Done means identifying whether the configuration is unsupported, ignored, or misconfigured and documenting the required change.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.