[docs] Document bucket rescaling for future partitions
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched the existing issues and found no dedicated documentation task for this feature.
### Description
Add user-facing documentation for bucket rescaling for future partitions, tracked by umbrella #4288 and implemented by #3907 / PR #3908.
The documentation should explain why users may want to increase or decrease the bucket count as workload changes, and how to use the feature safely.
### Documentation scope
- **Semantics and scope:** Explain the distinction between the table's default `bucket.num` and each partition's actual bucket count. A change applies only to partitions created afterward; existing partitions keep their original layout without data redistribution or lake-file rewrites. Clarify that a partition already pre-created for a future date is still an existing partition and is not affected by a later change.
- **Usage walkthrough:** Provide runnable Flink SQL examples that create a partitioned table, create and populate a partition, change `bucket.num`, and then create and populate another partition. Show both increasing and decreasing the count, the expected bucket counts of old and new partitions, and how to inspect those counts through a supported API or command. Demonstrate reads across the partitions and subsequent writes to an existing partition.
- **Partition creation modes:** Explain how the setting takes effect for explicit, automatic/pre-created, and dynamically created partitions where supported.
- **Constraints and operational guidance:** State supported table types, valid bucket-count limits, the release introducing the feature, and applicable client/connector and lake-format compatibility requirements. Explain the user-visible behavior of writes concurrent with a bucket-count change and any required recovery or restart procedure, based on the final implementation. Distinguish this feature from redistributing an existing partition or changing Flink job parallelism.
- **Discoverability:** Update the bucketing and partitioning guides, Flink DDL `ALTER TABLE ... SET` documentation, and the `bucket.num` option reference. Cross-link the relevant lake-format guidance where needed.
Suggested existing pages:
- `website/docs/table-design/data-distribution/bucketing.md`
- `website/docs/table-design/data-distribution/partitioning.md`
- `website/docs/engine-flink/ddl.md`
- `website/docs/engine-flink/options.md`
### Acceptance criteria
- A user can follow the examples to create partitions with different bucket counts and verify that older partitions remain unchanged.
- Examples and compatibility statements are checked against the target release; follow-up behavior tracked in #4287 is documented as available only when implemented.
- Terminology and links are consistent across the updated pages, and the documentation build and link checks pass.
### Willingness to contribute
- [ ] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the implementation in #3907 and PR #3908, then review the four suggested pages: bucketing.md, partitioning.md, ddl.md, and options.md. Run the documented Flink SQL examples against the target release and verify compatibility, partition bucket counts, cross-partition reads, and writes to existing partitions. Done means consistent guidance, links, examples, and passing documentation build and link checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100