Epic: partitioned namespace

Code Design

1. ManifestNamespace

• The ManifestNamespace will introduce an extended_properties feature, which supports storing specified properties as __manifest columns. These properties can be specified during create/describe table or namespace operations and are then set to or read from __manifest extended columns.
• ManifestNamespace provides a public insert_into_manifest method to facilitate transactional commits for PartitionedNamespace.

2. PartitionedNamespace
PartitionedNamespace is an impl of trait LanceNamespace, and has a member of ManifestNamespace. PartitionedNamespace exposes two categories of APIs:

• The first category inherits methods from LanceNamespace. All of their implementations directly delegate to ManifestNamespace.
• The second category consists of methods with partition semantics, such as plan_scan and resolve_or_create_partition_table. These methods encapsulate partition-related logic and are exposed to upper-layer engines for integration via public APIs.

Step by Step

• (lance-namespace) Add partition spec to rest.yaml. https://github.com/lance-format/lance-namespace/pull/297
• (lance-namespace) Add table properties. https://github.com/lance-format/lance-namespace/pull/299
• (lance) Add manifest namespace extended properties. https://github.com/lance-format/lance/pull/6215
• (fix) TooMuchWriteContention should be converted to a retryable namespace error https://github.com/lance-format/lance/pull/6177
• (lance) Multi-table transactions. https://github.com/lance-format/lance/pull/6173
• (lance) Add partitioned namespace. https://github.com/lance-format/lance/pull/5896
• (lance) Implement PartitionedNamespace. https://github.com/lance-format/lance/pull/5638

After PartitionedNamespace is implemented, we can start integrating with engines.

Related Links

• Discussion. https://github.com/lance-format/lance-namespace/discussions/272

Start by reviewing the unchecked Step by Step items, especially PRs #6215, #6177, #6173, #5896, and #5638, then read Discussion #272 for context. The issue is complete when the listed ManifestNamespace, retryability, transaction, and PartitionedNamespace work is implemented so engine integration can begin.