simple: add an opt-out for WATERMARK messages, symmetric with avro-enable-watermark

Is your feature request related to a problem?

WATERMARK messages cannot be disabled or slowed for the simple protocol, and their volume is a function of topology rather than of data volume — so a changefeed over many low-traffic tables pays for them indefinitely.

Measured on v8.5.1: watermarks are emitted once per second per topic-partition. On a 3-partition topic over a 45-second window each partition received 45, 46 and 47 watermarks respectively; on a 24-partition topic, 24/sec in aggregate. Inter-arrival deltas were exactly 1000 ms in 75 of 82 samples, matching the documented "the current interval is 1 second".

So the rate is topics × partitions messages/sec, independent of DML throughput. Under the default {schema}.{table} dispatcher that scales with table count, and it is driven by idle tables — a table with zero writes still emits one watermark per second per partition of its topic, forever.

As an illustration, a changefeed over 1,000 tables at 12 partitions each is 12,000 msg/sec, or roughly 1.04 billion messages/day. At the 86 bytes we measured per simple watermark that is about 89 GB/day of pure progress signalling, before replication factor. Consolidating to a single topic brings the same changefeed to 12/sec, which is the workaround we adopted — but it costs per-table retention, per-table partition budgeting and cheap single-table subscription, and it should not be the only way out.

There is currently no way to opt out. pkg/sink/codec/simple/encoder.go's EncodeCheckpointEvent marshals and returns a message unconditionally, with no config read, where the canal-json encoder's equivalent opens with an early return:

// canal_json_row_event_encoder.go
func (c *JSONRowEventEncoder) EncodeCheckpointEvent(ts uint64) (*common.Message, error) {
	if !c.config.EnableTiDBExtension {
		return nil, nil
	}

We enumerated the full SinkConfig TOML surface for release-8.5 — TiCDC helpfully rejects unknown keys (contained unknown configuration options: sink.<key>), so the list is exhaustive — and there is no watermark-related key for simple. avro-enable-watermark exists but is only consulted by the avro protocol, so protocol=simple&encoding-format=avro still emits them.

Describe the feature you'd like

An opt-out for simple, symmetric with the one avro already has.

The precedent is already in the codebase: avro-enable-watermark defaults to off, and pkg/sink/codec/avro/avro.go gates emission on it. So the concept, the naming pattern and the default are all settled — this is filling in an asymmetry rather than introducing anything new.

Concretely, either would work for us:

• simple-enable-watermark (bool, defaulting to on to preserve current behaviour) — closest match to the avro option, and the smaller change: one guard in EncodeCheckpointEvent.
• Or a cadence control, if a hard off is considered unsafe for consumers that rely on watermarks to advance resolved progress. Even a configurable interval would help, since the cost here is message count, not bytes.

We'd prefer the boolean, but a cadence knob addresses the volume problem just as well and may be easier to justify.

Describe alternatives you've considered

• Consolidating to one topic with more partitions. This is what we did, and it works — 24/sec at our layout. But it trades away per-table retention policies, per-table partition budgeting, and the ability for a consumer to subscribe to a single table cheaply. It is a real workaround, not a free one, and it shouldn't be the only lever.
• Filtering downstream. We filter non-DML records in the sink connector's transform chain, which protects the destination tables. It does not recover Kafka storage, network or broker throughput — those are spent at produce time.
• Switching to canal-json, whose watermarks are off by default. Not usable for us: claim-check on canal-json requires enable-tidb-extension=true, which turns TIDB_WATERMARK back on — and canal-json's watermark carries the full envelope with every field nulled, which we measured at 231 bytes against simple's 86.
• encoding-format=avro on simple, hoping avro-enable-watermark would apply. It does not; that option is only read by the avro protocol.

Teachability, Documentation, Adoption, Migration Strategy

Adoption is opt-in and the default preserves today's behaviour, so no migration concern. Documentation could mirror the existing avro-enable-watermark entry.

The guidance worth adding alongside it is the topology point, which we didn't find documented anywhere and which surprised us: watermark volume scales with topics × partitions, not with throughput. That is the single most useful thing to know when choosing a dispatcher, and it is currently something users have to discover by measurement.

Lower priority than #6106 from our side — we have a working topology workaround, whereas the start_ts gap has none. Filing it separately for that reason, and at Ben Sherrill's suggestion.

Happy to test a build; our v8.5.1 harness already counts non-DML records per partition, so verifying the option is straightforward.

Start with pkg/sink/codec/simple/encoder.go and compare its EncodeCheckpointEvent behavior with the avro gate in pkg/sink/codec/avro/avro.go. Trace the existing SinkConfig option and documentation entries for avro-enable-watermark. Done means simple watermark emission is configurable, the default preserves current behavior, and the setting is documented and covered by the relevant tests.