apache / apache/fluss

[docs] Document local log TTL and active segment rolling

Closed
#3,938 1 comment 0 reactions 0 assignees View on GitHub
priority=critical
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Motivation

Local log retention behavior is being extended by #3748 and #3830. The user-facing documentation should describe the resulting configuration model and clearly distinguish remote log retention, local log retention, and active-segment rolling. It should also provide safe rollout guidance for clusters upgrading from 0.9 to 1.0.

### Documentation scope

- Document `table.log.local-ttl`:
- it controls retention of local log segments;
- a local segment is eligible for TTL cleanup only after it has been copied to remote storage;
- document its default value and validation constraints, including its relationship with `table.log.ttl`;
- reconcile with the initial documentation changes already included in #3778.
- Document `log.retention.roll-active-segment.enabled`:
- it is a server configuration and is disabled by default;
- when enabled, a non-empty active segment may be rolled after the table log TTL expires and the high watermark reaches the log end offset;
- explain that rolling makes the segment eligible for remote upload and subsequent local cleanup.
- Expand the TTL page to distinguish:
- `table.log.ttl` for table/remote log retention;
- `table.log.local-ttl` for copied local segments;
- active-segment rolling controlled by `log.retention.roll-active-segment.enabled`.
- Update or regenerate the relevant table and server configuration reference pages.
- Update the tiered-storage documentation and add cross-links to the TTL page.
- Update the 1.0 upgrade notes for clusters upgrading from 0.9:
- keep `log.retention.roll-active-segment.enabled` disabled during the upgrade; this is also the default value;
- after all cluster components have been upgraded to 1.0 and the upgrade is complete, enable the option through a dynamic cluster configuration update;
- explain that enabling the option allows expired active local log segments to be rolled and subsequently cleaned up automatically.

### Related issues and pull requests

- Related to #3748
- Related to #3830
- Related PR: #3778
- Related PR: #3831

### Acceptance criteria

- Both configuration keys are searchable in the published documentation.
- Their defaults, constraints, scope, and interaction are documented.
- The TTL page explains the local-segment lifecycle from active segment, to rolled/uploaded segment, to local cleanup.
- The 1.0 upgrade notes document the staged 0.9-to-1.0 rollout: keep `log.retention.roll-active-segment.enabled` disabled during the upgrade, then enable it with a dynamic cluster configuration update after the upgrade is complete.
- Existing documentation changes in #3778 are retained and reconciled instead of duplicated.
- The TTL, configuration-reference, tiered-storage, and upgrade-note pages link to each other where appropriate.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the TTL, table and server configuration reference, tiered-storage, and 1.0 upgrade-note pages, then compare the existing changes in #3778 and related work in #3831. Document both configuration keys, their defaults, constraints, scope, interactions, and the local-segment lifecycle, with cross-links and the staged upgrade guidance. Done means both keys are searchable in published documentation and the acceptance criteria are met without duplicated content.

Written by the indexing model from the issue text.

Assessment

Domain
documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.