Generic Tables cannot store or expose structured schema metadata
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 137
Description
### Describe the bug
Polaris Generic Tables can register non-Iceberg table formats, but their API model has no first-class schema field. The create/load payload only exposes `name`, `format`, `base-location`, `doc`, and string `properties`.
As a result, a catalog integration cannot populate ordered columns, types, or required/nullability metadata for a Generic Table. Clients and catalog UIs can render a structured schema for Iceberg tables from `metadata.schemas`, but equivalent metadata cannot be represented for Generic Tables.
### To Reproduce
1. Create a Generic Table for any non-Iceberg format.
2. Include its schema as JSON in a string property because no schema field exists.
3. Load the Generic Table through the Polaris API.
4. Observe that schema information is available only as an opaque property; there is no structured schema in the response and no schema update operation.
Example workaround property:
```json
{
"external.schema-json": "{\"columns\":[{\"name\":\"customer_id\",\"type\":\"long\",\"nullable\":true}]}"
}
```
### Actual Behavior
Column metadata must be serialized into arbitrary string properties. It is not validated, typed, queryable, interoperable, or exposed through the same schema model used for Iceberg tables. A schema change also requires replacing the Generic Table metadata rather than updating a structured schema.
### Expected Behavior
Generic Tables should support an optional structured schema in their create/load/update API, including:
- ordered field names;
- field types, including nested types;
- required/nullability information; and
- schema evolution or replacement.
This would let non-Iceberg formats participate as proper catalog tables rather than name/location registrations with opaque properties.
### Additional context
This affects any integration cataloging external formats whose physical system remains authoritative for schema. Encoding schema JSON in properties preserves information but does not provide catalog-level schema semantics.
### System information
Polaris Catalog Version: 1.5.0
Setup: Generic Table backed by an external non-Iceberg system
Contributor guide
Research direction
Start by tracing the Generic Table create, load, and update API models described in the issue, then compare them with the Iceberg metadata.schemas representation. Define how ordered fields, nested types, nullability, and schema replacement should be represented and validated. Done means Generic Tables can expose structured schema metadata and support the specified schema evolution behavior through the API.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend-api-design, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100