Ugbot / Ugbot/Agentic-Streaming

Paradigm completion: fault tolerance and recovery on the Flink runtime

Open
#6 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
37
Forks
2
Avg merge
10h 46m
Merged PRs (30d)
28

Description

The bar: state as a materialized view over the runtime own ordered log plus checkpoints — a separate durable event log is not required.

Two runtimes clear it. agentic-pekko uses real Pekko Persistence (EventSourcedBehavior + journal) and has a test proving the transcript survives entity restart while brain invocations stay flat. agentic-clj uses real Datomic with working as-of time travel.

The flagship Flink runtime does not. enableCheckpointing, CheckpointingMode, setStateBackend and CheckpointedFunction appear zero times in the framework — only in a build-excluded demo, and there with the non-durable HashMapStateBackend.

Supporting gaps:

  • PollingSource splits are explicitly stateless (serialize() returns new byte[0]), so every non-Kafka channel is at-most-once across a restart.
  • No sink implements SupportsCommitter/TwoPhaseCommittingSink.
  • Savepoint restore of an A2A bridge stream NPEs (snapshot restored with a null serializer arg).
  • Both Flink-state vector memories call getMapState() from non-keyed ProcessFunctions, so they throw in open() and are unreachable through the shipped corpus/retrieval API.

Deliverable: checkpointing configured (or explicitly documented as a caller responsibility), a durable state backend option, sources that actually snapshot offsets, and a test that restarts from a checkpoint and asserts the conversation view is intact.


Roadmap epic. Detailed findings, file:line citations and sub-tasks live in the AI-grind tracker (project AGS), which is the source of truth for this work. Local dev — build, test, tracking, profiling — is managed with AI-grind.


Tracked as AGS-6 in the devtools-mcp tracker.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The AI-grind tracker project AGS-6 is the stated source for file and line citations; start there, then inspect the Flink runtime entry points named in the issue, including PollingSource, checkpoint/savepoint handling, sink commit interfaces, and vector-memory open paths. Done means checkpoint recovery preserves the conversation view, source offsets are snapshotted, a durable backend option exists, and the checkpoint restart test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.