Tracking Kafka enhancement and migrate to franz-go

This issue tracks the complete Kafka Sink migration from Sarama to franz-go. It is the roadmap and dependency index; detailed implementation and acceptance criteria belong to the linked task issues.

Goal and invariants

• Plan and implement from master; the existing franz-go POC is reference material, not a branch to merge directly.
• Replace only the Kafka client layer. Preserve Kafka Sink configuration, routing, message, error, retry, callback, checkpoint, and resource-lifecycle behavior unless a separately reviewed change says otherwise.
• Define TiCDC behavior through client-independent tests over TiCDC-owned interfaces. Sarama and franz-go adapter tests may verify client-specific mapping and cleanup, but cannot be the only definition of Kafka Sink behavior.
• Complete the migration in four milestones: establish the test baseline, introduce franz-go and make it the default with a Sarama fallback, validate production readiness, then remove Sarama.

Execution order

1. Complete all Milestone 0 baseline tasks.
2. Merge and validate the franz-go implementation in Milestone 1.
3. Complete staged production validation in Milestone 2.
4. Remove Sarama and migration-only code in Milestone 3.

The dependency chain is:

client-independent baseline → franz-go + Sarama fallback → production validation → Sarama removal

Milestone 0: establish the client-independent test baseline

Task issues

• #5884 — decouple Kafka Sink tests from Sarama.
• #5887 — closed as not planned after its configuration and authentication scope was absorbed by #5884 and #5892; no separate work remains under this issue.
• #5888 — closed as not planned after its broker-mock cleanup scope was absorbed by #5884; no separate work remains under this issue.
• #5892 — decouple security SASL types from Sarama.
• #5885 — decouple Admin and Topic Manager tests from Sarama.
• #5886 — define client-independent producer behavior contracts.
• #5889 — make common behavior tests and client-adapter tests explicitly separate.

Closed or superseded task issues remain in this list so that the tracker preserves the complete scope and disposition history.

Exit criteria

• Common Kafka Sink behavior tests use only TiCDC-owned interfaces and types.
• Client-specific tests and mocks remain explicitly adapter-only.

Milestone 1: introduce franz-go and make it the default

Task issues

• #832 — replace the fragile Kafka client path with a franz-go implementation while retaining Sarama as a per-Changefeed fallback during migration.
• #5882 — migrate the integration-test Kafka topic helper from Sarama to franz-go while preserving its existing CLI.
• Migrate the Kafka consumer from confluentic-kafka-go to franz-go

Required validation

• Run the same Admin, sync producer, async producer, configuration, error, and lifecycle contract tests against both clients without duplicating testcase logic.
• Verify that Verify and runtime Sink creation select the same client and that different Changefeeds can use different clients in one process.
• Validate supported Kafka versions, TLS, mTLS, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512, GSSAPI, OAUTHBEARER, invalid credentials, and minimum ACLs.
• Run the reusable regression and failure-recovery suites with franz-go as the default and Sarama as the explicit fallback.
• Validate message-size boundaries, retry and ordering guarantees, callback/checkpoint correctness, bounded producer memory, resource cleanup, logs, metrics, and Grafana dashboards.
• Compare franz-go and Sarama throughput, latency, CPU, memory, GC, goroutines, and retry behavior in equivalent environments.

Exit criteria

• franz-go is the default Kafka client for PR, Nightly, and Release validation.
• Sarama remains selectable per Changefeed as a tested fallback.
• No release-blocking correctness, compatibility, authentication, stability, resource, performance, or observability issue remains.

Milestone 2: staged production validation

Create and link a dedicated rollout issue before this milestone begins.

Required work

• Deploy franz-go progressively to representative production workloads and Kafka environments.
• Validate correctness, stability, recovery, performance, resource usage, authentication, and observability over an agreed observation window.
• Confirm that production failures are diagnosable through logs, metrics, and dashboards.
• Exercise the per-Changefeed Sarama fallback and version rollback procedure.

Exit criteria

• franz-go has operated as the default implementation in key production scenarios without data-correctness failures or unexplained stability/performance regressions.
• Monitoring and rollback procedures have been validated.
• The team has an explicit decision to remove the Sarama fallback.

Milestone 3: remove Sarama and migration-only code

• #5890 — starts only after Milestone 2 approval and removes the Sarama implementation, fallback selector, adapter-only tests, generated mocks, dependencies, metrics, and migration-only compatibility code.

Exit criteria

• Kafka Sink production code, tests, test tooling, dependencies, and monitoring no longer reference Sarama.
• franz-go retains equivalent user-visible configuration, error, lifecycle, and regression coverage.
• Repository checks, race-enabled unit tests, Kafka integration tests, and relevant test-infra plans pass.