Azure / Azure/azure-sdk-for-java

[Cosmos Spark] Add missing container property support in Spark Catalog

Open
#48,360 1 comment 0 reactions 1 assignee Claimed by @xinlian12 View on GitHub
Client Cosmos cosmos:v4-item ffcf-ga needs-team-attention Service Attention
Dominant language
Java
Stars
2.6k
Forks
2.2k
Avg merge
2d 8h
Merged PRs (30d)
178

Description

## Summary

The Spark Catalog currently supports a subset of Cosmos DB container properties when creating containers via `TBLPROPERTIES`. This issue tracks adding support for the remaining properties.

Related PR: #48349 (added Vector Embedding Policy support)

## Currently Supported

| Property | TBLPROPERTIES Key |
|---|---|
| Partition Key Definition | `partitionKeyPath` |
| Indexing Policy | `indexingPolicy` |
| Default TTL | `defaultTtlInSeconds` |
| Analytical Store TTL | `analyticalStoreTtlInSeconds` |
| Vector Embedding Policy | `vectorEmbeddingPolicy` (added in #48349) |

## Missing Properties

- [ ] **Unique Key Policy** (`UniqueKeyPolicy`) — enforce unique constraints on document fields
- [ ] **Conflict Resolution Policy** (`ConflictResolutionPolicy`) — configure conflict resolution for multi-region writes
- [ ] **Computed Properties** (`ComputedProperty`) — define server-side computed columns
- [ ] **Client Encryption Policy** (`ClientEncryptionPolicy`) — configure field-level encryption
- [ ] **Full Text Policy** (`CosmosFullTextPolicy`) — configure full text search
- [ ] **Change Feed Policy** (`ChangeFeedPolicy`) — configure change feed behavior

## Implementation Notes

Each property follows the same pattern established by the Vector Embedding Policy PR (#48349):

1. Add a constant in `CosmosConstants.scala` (`TableProperties`)
2. Add a getter in `CosmosContainerProperties.scala`
3. Add JSON bridge methods in `SparkModelBridgeInternal.scala`
4. Wire into container creation in `CosmosCatalogCosmosSDKClient.scala` and `CosmosCatalogManagementSDKClient.scala`
5. Add integration tests in `CosmosCatalogITestBase.scala`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.