apache / apache/fluss

[docs] Document bucket rescaling for future partitions

Closed
#4,289 0 comments 0 reactions 0 assignees View on GitHub
priority=critical
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched the existing issues and found no dedicated documentation task for this feature.

### Description

Add user-facing documentation for bucket rescaling for future partitions, tracked by umbrella #4288 and implemented by #3907 / PR #3908.

The documentation should explain why users may want to increase or decrease the bucket count as workload changes, and how to use the feature safely.

### Documentation scope

- **Semantics and scope:** Explain the distinction between the table's default `bucket.num` and each partition's actual bucket count. A change applies only to partitions created afterward; existing partitions keep their original layout without data redistribution or lake-file rewrites. Clarify that a partition already pre-created for a future date is still an existing partition and is not affected by a later change.
- **Usage walkthrough:** Provide runnable Flink SQL examples that create a partitioned table, create and populate a partition, change `bucket.num`, and then create and populate another partition. Show both increasing and decreasing the count, the expected bucket counts of old and new partitions, and how to inspect those counts through a supported API or command. Demonstrate reads across the partitions and subsequent writes to an existing partition.
- **Partition creation modes:** Explain how the setting takes effect for explicit, automatic/pre-created, and dynamically created partitions where supported.
- **Constraints and operational guidance:** State supported table types, valid bucket-count limits, the release introducing the feature, and applicable client/connector and lake-format compatibility requirements. Explain the user-visible behavior of writes concurrent with a bucket-count change and any required recovery or restart procedure, based on the final implementation. Distinguish this feature from redistributing an existing partition or changing Flink job parallelism.
- **Discoverability:** Update the bucketing and partitioning guides, Flink DDL `ALTER TABLE ... SET` documentation, and the `bucket.num` option reference. Cross-link the relevant lake-format guidance where needed.

Suggested existing pages:

- `website/docs/table-design/data-distribution/bucketing.md`
- `website/docs/table-design/data-distribution/partitioning.md`
- `website/docs/engine-flink/ddl.md`
- `website/docs/engine-flink/options.md`

### Acceptance criteria

- A user can follow the examples to create partitions with different bucket counts and verify that older partitions remain unchanged.
- Examples and compatibility statements are checked against the target release; follow-up behavior tracked in #4287 is documented as available only when implemented.
- Terminology and links are consistent across the updated pages, and the documentation build and link checks pass.

### Willingness to contribute

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the implementation in #3907 and PR #3908, then review the four suggested pages: bucketing.md, partitioning.md, ddl.md, and options.md. Run the documented Flink SQL examples against the target release and verify compatibility, partition bucket counts, cross-partition reads, and writes to existing partitions. Done means consistent guidance, links, examples, and passing documentation build and link checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
data-engineering, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.