apache / apache/polaris

Generic Tables cannot store or expose structured schema metadata

Open
#5,128 3 comments 0 reactions 0 assignees View on GitHub
stale
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

Polaris Generic Tables can register non-Iceberg table formats, but their API model has no first-class schema field. The create/load payload only exposes `name`, `format`, `base-location`, `doc`, and string `properties`.

As a result, a catalog integration cannot populate ordered columns, types, or required/nullability metadata for a Generic Table. Clients and catalog UIs can render a structured schema for Iceberg tables from `metadata.schemas`, but equivalent metadata cannot be represented for Generic Tables.

### To Reproduce

1. Create a Generic Table for any non-Iceberg format.
2. Include its schema as JSON in a string property because no schema field exists.
3. Load the Generic Table through the Polaris API.
4. Observe that schema information is available only as an opaque property; there is no structured schema in the response and no schema update operation.

Example workaround property:

```json
{
"external.schema-json": "{\"columns\":[{\"name\":\"customer_id\",\"type\":\"long\",\"nullable\":true}]}"
}
```

### Actual Behavior

Column metadata must be serialized into arbitrary string properties. It is not validated, typed, queryable, interoperable, or exposed through the same schema model used for Iceberg tables. A schema change also requires replacing the Generic Table metadata rather than updating a structured schema.

### Expected Behavior

Generic Tables should support an optional structured schema in their create/load/update API, including:

- ordered field names;
- field types, including nested types;
- required/nullability information; and
- schema evolution or replacement.

This would let non-Iceberg formats participate as proper catalog tables rather than name/location registrations with opaque properties.

### Additional context

This affects any integration cataloging external formats whose physical system remains authoritative for schema. Encoding schema JSON in properties preserves information but does not provide catalog-level schema semantics.

### System information

Polaris Catalog Version: 1.5.0
Setup: Generic Table backed by an external non-Iceberg system

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Generic Table create, load, and update API models described in the issue, then compare them with the Iceberg metadata.schemas representation. Define how ordered fields, nested types, nullability, and schema replacement should be represented and validated. Done means Generic Tables can expose structured schema metadata and support the specified schema evolution behavior through the API.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.