apache / apache/gravitino

[Improvement] Fix policy version limit being applied per policy instead of globally

Open
#10,275 4 comments 0 reactions 1 assignee Claimed by @marsii1017 View on GitHub
good first issue improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

### What would you like to be improved?

PolicyMetaService.deletePolicyVersionsByRetentionCount(versionRetentionCount, limit) applies limit inside a loop for each policy. In production, this is called by the relational garbage collector with GARBAGE_COLLECTOR_SINGLE_DELETION_LIMIT (100). With multiple policies, one GC iteration can delete up to limit * numberOfPolicies, exceeding the intended single-batch cap and causing unexpectedly large cleanup operations.

### How should we improve?

Enforce limit as a global budget per method call. Track remaining = limit, and for each policy call softDeletePolicyVersionsByRetentionLine(..., remaining), subtracting deleted rows from remaining; stop when remaining <= 0. This preserves bounded batch behavior while still progressing cleanup across runs.

here a test to help:
```
@TestTemplate
public void testDeletePolicyVersionsByRetentionCountRespectGlobalLimit() throws IOException {
BaseMetalake metalake = createAndInsertMakeLake(METALAKE_NAME + "_global_limit");

PolicyEntity policy1 =
createPolicy(
RandomIdGenerator.INSTANCE.nextId(),
NamespaceUtil.ofPolicy(metalake.name()),
"policy_limit_1",
AUDIT_INFO);
PolicyEntity policy2 =
createPolicy(
RandomIdGenerator.INSTANCE.nextId(),
NamespaceUtil.ofPolicy(metalake.name()),
"policy_limit_2",
AUDIT_INFO);
backend.insert(policy1, false);
backend.insert(policy2, false);

PolicyEntity policy1V2 =
PolicyEntity.builder()
.withId(policy1.id())
.withNamespace(policy1.namespace())
.withName(policy1.name())
.withPolicyType(policy1.policyType())
.withComment(policy1.comment())
.withEnabled(!policy1.enabled())
.withContent(policy1.content())
.withAuditInfo(AUDIT_INFO)
.build();
PolicyEntity policy2V2 =
PolicyEntity.builder()
.withId(policy2.id())
.withNamespace(policy2.namespace())
.withName(policy2.name())
.withPolicyType(policy2.policyType())
.withComment(policy2.comment())
.withEnabled(!policy2.enabled())
.withContent(policy2.content())
.withAuditInfo(AUDIT_INFO)
.build();
backend.update(policy1.nameIdentifier(), Entity.EntityType.POLICY, e -> policy1V2);
backend.update(policy2.nameIdentifier(), Entity.EntityType.POLICY, e -> policy2V2);

int deletedCount = PolicyMetaService.getInstance().deletePolicyVersionsByRetentionCount(1L, 1);

assertEquals(1, deletedCount);
}

```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.