apache / apache/polaris

Policy and generic-table list endpoints ignore page-token and page-size

Open
#5,311 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

Three list endpoints accept `page-token` and `page-size` and declare
`next-page-token` in their response schema, but the implementations ignore all
three. They always list everything and never return a continuation token, so a
client that paginates cannot tell a complete result from a truncated one, and
the response size is unbounded.

| Endpoint | Honors pagination |
| --- | --- |
| `listNamespaces`, `listTables`, `listViews` | yes |
| `listSemanticModels` | yes |
| `listPolicies` | **no** |
| `getApplicablePolicies` | **no** |
| `listGenericTables` | **no** |

The parameters reach the adapter and are then dropped:

- `PolicyCatalogAdapter#listPolicies` / `#getApplicablePolicies` — `pageToken`
and `pageSize` are never referenced in the method body; `PolicyCatalog#listPolicies`
calls `metaStoreManager.listEntities(..., PageToken.readEverything())`.
- `GenericTableCatalogAdapter#listGenericTables` — same, delegating to
`GenericTableCatalogHandler#listGenericTables(Namespace)`, which takes no page token.

The specifications declare the parameters and the response field:

- `spec/polaris-catalog-apis/policy-apis.yaml` — `ListPoliciesResponse` and
`GetApplicablePoliciesResponse` both declare `next-page-token`, and both
operations reference the shared `page-token` / `page-size` parameters.
- `spec/polaris-catalog-apis/generic-tables-api.yaml` — `ListGenericTablesResponse`
declares `next-page-token`.

`listSemanticModels` already implements the intended pattern end to end
(`SemanticModelCatalog#listSemanticModels` threads a `PageToken` into
`listEntities` and sets `next-page-token` from `result.getPage().encodedResponseToken()`),
so the persistence layer needs no changes.

### To Reproduce

1. Create a namespace and several policies in it.
2. `GET /polaris/v1/{prefix}/namespaces/{namespace}/policies?pageSize=1`
3. All policies are returned and `next-page-token` is absent, instead of one
policy plus a continuation token.

The same applies to `GET /polaris/v1/{prefix}/applicable-policies?pageSize=1`
and `GET /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables?pageSize=1`.

### Actual Behavior

`page-size` and `page-token` are ignored, the full result set is returned in one
response, and `next-page-token` is never populated.

### Expected Behavior

These endpoints honor `page-size` / `page-token` and return `next-page-token`
when more results remain, consistent with `listNamespaces`, `listTables`,
`listViews` and `listSemanticModels`.

### Additional context

Two points that likely need a decision before anyone writes code, which is why
this is filed as an issue rather than opened as a pull request:

1. **`listGenericTables` requires changing a public extension point.**
`GenericTableCatalog#listGenericTables(Namespace)` lives in `polaris-core`, so
adding a `PageToken` parameter changes an SPI. Per CONTRIBUTING.md that needs
discussion on the dev mailing list first. Should this part be split out and
taken to dev@ separately from the two policy endpoints?

2. **Result ordering.** `PolicyCatalogHandler` returns identifiers as a
`HashSet` and `GenericTableCatalogHandler` as a `LinkedHashSet`. Stable
pagination needs a deterministic order, so the response assembly has to change
alongside the token plumbing. The schemas declare `uniqueItems: true`, so the
set semantics themselves can stay.

`getApplicablePolicies` is the awkward one of the three: it computes an effective
policy set by walking the entity hierarchy in memory rather than performing a
single backing-store listing, so paginating it is not simply a matter of passing
a token through. It may be reasonable to address the two straightforward
endpoints first and handle that one separately.

I am happy to work on this once there is agreement on the scope and on the SPI
question.

### System information

Observed on `main` at commit 3bf4126b5.

Contributor guide

Open the contributing guide

Research direction

Start by comparing the existing pagination flow in SemanticModelCatalog#listSemanticModels with PolicyCatalogAdapter, PolicyCatalog, and GenericTableCatalogAdapter#listGenericTables. Read policy-apis.yaml and generic-tables-api.yaml, then resolve the SPI and ordering questions before changing scope. Done means the selected endpoints honor page-size and page-token and return next-page-token when results remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.