apache / apache/polaris

Python client drops external catalog subtype fields during deserialization

Open
#5,409 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 17h
Merged PRs (30d)
137

Description

### Apache Polaris version

1.7.0. The same schema shape is present on current `main` at `ba152c04f7f985bbaeb487a55ef442df52321afe`.

### Operating system and architecture

macOS 26.6.2, arm64; Python 3.14.6; Pydantic 2.13.4.

### What did you do?

Deserialize management API payloads for each external-catalog connection subtype through the generated base model, as the Python API client does for `ExternalCatalog.connectionConfigInfo`:

```python
from apache_polaris.sdk.management.models.connection_config_info import (
ConnectionConfigInfo,
)
from apache_polaris.sdk.management.models.iceberg_rest_connection_config_info import (
IcebergRestConnectionConfigInfo,
)

cases = [
(
{
"connectionType": "ICEBERG_REST",
"uri": "https://example.invalid",
"remoteCatalogName": "prod",
},
"remote_catalog_name",
),
(
{
"connectionType": "HADOOP",
"uri": "file:///warehouse",
"warehouse": "/warehouse",
},
"warehouse",
),
(
{
"connectionType": "HIVE",
"uri": "thrift://example.invalid:9083",
"warehouse": "/warehouse",
},
"warehouse",
),
]

for payload, field in cases:
parsed = ConnectionConfigInfo.from_dict(payload)
print(payload["connectionType"], getattr(parsed, field))

# Control: direct subtype validation retains the same field.
control = IcebergRestConnectionConfigInfo.model_validate(cases[0][0])
print("control", control.remote_catalog_name)
```

This reproduces with the published `apache-polaris==1.7.0` wheel and with a client generated from current `main`.

### What did you expect to see?

The discriminator should select the concrete connection model and retain all subtype-specific fields:

```text
ICEBERG_REST prod
HADOOP /warehouse
HIVE /warehouse
control prod
```

### What did you see instead?

The generated base model selects the correct subtype but silently drops its locator field:

```text
ICEBERG_REST None
HADOOP None
HIVE None
control prod
```

Inherited fields such as `uri` remain populated. Direct validation of the concrete subtype also retains the field, which isolates the failure to generated polymorphic deserialization.

### Impact

The setup exporter reads `remote_catalog_name` and `warehouse` from these deserialized models. Consequently, an exported external-catalog configuration can omit the remote catalog name or warehouse and cannot reliably recreate the original federation topology. The same data loss affects Python SDK users reading external catalog responses.

### Likely root cause

The three connection subtypes declare `properties` as a sibling of `allOf`:

```yaml
IcebergRestConnectionConfigInfo:
type: object
allOf:
- $ref: '#/components/schemas/ConnectionConfigInfo'
properties:
remoteCatalogName:
type: string
```

OpenAPI Generator emits the subtype attribute but does not include it in the generated discriminator `from_dict` mapping. Polaris storage subtypes already use a generator-compatible pattern where subtype properties are a second object inside `allOf`.

### Proposed fix

Move the subtype-specific `properties` for `IcebergRestConnectionConfigInfo`, `HadoopConnectionConfigInfo`, and `HiveConnectionConfigInfo` into a second `type: object` member of `allOf`, matching `AwsStorageConfigInfo`. Regenerate the client and cover discriminator-based deserialization with a focused unit test. This does not change JSON field names or wire semantics.

### Acceptance criteria

- `ConnectionConfigInfo.from_dict` retains `remoteCatalogName` for `ICEBERG_REST`.
- `ConnectionConfigInfo.from_dict` retains `warehouse` for `HADOOP` and `HIVE`.
- A setup export from API-deserialized models preserves these values.
- Generated clients, Python unit tests, lint, build, and repository-required checks pass.

Contributor guide

Open the contributing guide

Research direction

Start with ConnectionConfigInfo.from_dict and the schema definitions for IcebergRestConnectionConfigInfo, HadoopConnectionConfigInfo, and HiveConnectionConfigInfo, comparing their allOf layout with AwsStorageConfigInfo. Regenerate the Python client and add focused discriminator-deserialization coverage for remoteCatalogName and warehouse; done when setup export preserves those values and the required client, unit, lint, build, and repository checks pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi, python
Domain
api, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.