Policy and generic-table list endpoints ignore page-token and page-size
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Three list endpoints accept `page-token` and `page-size` and declare
`next-page-token` in their response schema, but the implementations ignore all
three. They always list everything and never return a continuation token, so a
client that paginates cannot tell a complete result from a truncated one, and
the response size is unbounded.
| Endpoint | Honors pagination |
| --- | --- |
| `listNamespaces`, `listTables`, `listViews` | yes |
| `listSemanticModels` | yes |
| `listPolicies` | **no** |
| `getApplicablePolicies` | **no** |
| `listGenericTables` | **no** |
The parameters reach the adapter and are then dropped:
- `PolicyCatalogAdapter#listPolicies` / `#getApplicablePolicies` — `pageToken`
and `pageSize` are never referenced in the method body; `PolicyCatalog#listPolicies`
calls `metaStoreManager.listEntities(..., PageToken.readEverything())`.
- `GenericTableCatalogAdapter#listGenericTables` — same, delegating to
`GenericTableCatalogHandler#listGenericTables(Namespace)`, which takes no page token.
The specifications declare the parameters and the response field:
- `spec/polaris-catalog-apis/policy-apis.yaml` — `ListPoliciesResponse` and
`GetApplicablePoliciesResponse` both declare `next-page-token`, and both
operations reference the shared `page-token` / `page-size` parameters.
- `spec/polaris-catalog-apis/generic-tables-api.yaml` — `ListGenericTablesResponse`
declares `next-page-token`.
`listSemanticModels` already implements the intended pattern end to end
(`SemanticModelCatalog#listSemanticModels` threads a `PageToken` into
`listEntities` and sets `next-page-token` from `result.getPage().encodedResponseToken()`),
so the persistence layer needs no changes.
### To Reproduce
1. Create a namespace and several policies in it.
2. `GET /polaris/v1/{prefix}/namespaces/{namespace}/policies?pageSize=1`
3. All policies are returned and `next-page-token` is absent, instead of one
policy plus a continuation token.
The same applies to `GET /polaris/v1/{prefix}/applicable-policies?pageSize=1`
and `GET /polaris/v1/{prefix}/namespaces/{namespace}/generic-tables?pageSize=1`.
### Actual Behavior
`page-size` and `page-token` are ignored, the full result set is returned in one
response, and `next-page-token` is never populated.
### Expected Behavior
These endpoints honor `page-size` / `page-token` and return `next-page-token`
when more results remain, consistent with `listNamespaces`, `listTables`,
`listViews` and `listSemanticModels`.
### Additional context
Two points that likely need a decision before anyone writes code, which is why
this is filed as an issue rather than opened as a pull request:
1. **`listGenericTables` requires changing a public extension point.**
`GenericTableCatalog#listGenericTables(Namespace)` lives in `polaris-core`, so
adding a `PageToken` parameter changes an SPI. Per CONTRIBUTING.md that needs
discussion on the dev mailing list first. Should this part be split out and
taken to dev@ separately from the two policy endpoints?
2. **Result ordering.** `PolicyCatalogHandler` returns identifiers as a
`HashSet` and `GenericTableCatalogHandler` as a `LinkedHashSet`. Stable
pagination needs a deterministic order, so the response assembly has to change
alongside the token plumbing. The schemas declare `uniqueItems: true`, so the
set semantics themselves can stay.
`getApplicablePolicies` is the awkward one of the three: it computes an effective
policy set by walking the entity hierarchy in memory rather than performing a
single backing-store listing, so paginating it is not simply a matter of passing
a token through. It may be reasonable to address the two straightforward
endpoints first and handle that one separately.
I am happy to work on this once there is agreement on the scope and on the SPI
question.
### System information
Observed on `main` at commit 3bf4126b5.
Contributor guide
Research direction
Start by comparing the existing pagination flow in SemanticModelCatalog#listSemanticModels with PolicyCatalogAdapter, PolicyCatalog, and GenericTableCatalogAdapter#listGenericTables. Read policy-apis.yaml and generic-tables-api.yaml, then resolve the SPI and ordering questions before changing scope. Done means the selected endpoints honor page-size and page-token and return next-page-token when results remain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend-api-design
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100