Python client drops external catalog subtype fields during deserialization
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 522
- Avg merge
- 1d 17h
- Merged PRs (30d)
- 137
Description
### Apache Polaris version
1.7.0. The same schema shape is present on current `main` at `ba152c04f7f985bbaeb487a55ef442df52321afe`.
### Operating system and architecture
macOS 26.6.2, arm64; Python 3.14.6; Pydantic 2.13.4.
### What did you do?
Deserialize management API payloads for each external-catalog connection subtype through the generated base model, as the Python API client does for `ExternalCatalog.connectionConfigInfo`:
```python
from apache_polaris.sdk.management.models.connection_config_info import (
ConnectionConfigInfo,
)
from apache_polaris.sdk.management.models.iceberg_rest_connection_config_info import (
IcebergRestConnectionConfigInfo,
)
cases = [
(
{
"connectionType": "ICEBERG_REST",
"uri": "https://example.invalid",
"remoteCatalogName": "prod",
},
"remote_catalog_name",
),
(
{
"connectionType": "HADOOP",
"uri": "file:///warehouse",
"warehouse": "/warehouse",
},
"warehouse",
),
(
{
"connectionType": "HIVE",
"uri": "thrift://example.invalid:9083",
"warehouse": "/warehouse",
},
"warehouse",
),
]
for payload, field in cases:
parsed = ConnectionConfigInfo.from_dict(payload)
print(payload["connectionType"], getattr(parsed, field))
# Control: direct subtype validation retains the same field.
control = IcebergRestConnectionConfigInfo.model_validate(cases[0][0])
print("control", control.remote_catalog_name)
```
This reproduces with the published `apache-polaris==1.7.0` wheel and with a client generated from current `main`.
### What did you expect to see?
The discriminator should select the concrete connection model and retain all subtype-specific fields:
```text
ICEBERG_REST prod
HADOOP /warehouse
HIVE /warehouse
control prod
```
### What did you see instead?
The generated base model selects the correct subtype but silently drops its locator field:
```text
ICEBERG_REST None
HADOOP None
HIVE None
control prod
```
Inherited fields such as `uri` remain populated. Direct validation of the concrete subtype also retains the field, which isolates the failure to generated polymorphic deserialization.
### Impact
The setup exporter reads `remote_catalog_name` and `warehouse` from these deserialized models. Consequently, an exported external-catalog configuration can omit the remote catalog name or warehouse and cannot reliably recreate the original federation topology. The same data loss affects Python SDK users reading external catalog responses.
### Likely root cause
The three connection subtypes declare `properties` as a sibling of `allOf`:
```yaml
IcebergRestConnectionConfigInfo:
type: object
allOf:
- $ref: '#/components/schemas/ConnectionConfigInfo'
properties:
remoteCatalogName:
type: string
```
OpenAPI Generator emits the subtype attribute but does not include it in the generated discriminator `from_dict` mapping. Polaris storage subtypes already use a generator-compatible pattern where subtype properties are a second object inside `allOf`.
### Proposed fix
Move the subtype-specific `properties` for `IcebergRestConnectionConfigInfo`, `HadoopConnectionConfigInfo`, and `HiveConnectionConfigInfo` into a second `type: object` member of `allOf`, matching `AwsStorageConfigInfo`. Regenerate the client and cover discriminator-based deserialization with a focused unit test. This does not change JSON field names or wire semantics.
### Acceptance criteria
- `ConnectionConfigInfo.from_dict` retains `remoteCatalogName` for `ICEBERG_REST`.
- `ConnectionConfigInfo.from_dict` retains `warehouse` for `HADOOP` and `HIVE`.
- A setup export from API-deserialized models preserves these values.
- Generated clients, Python unit tests, lint, build, and repository-required checks pass.
Contributor guide
Research direction
Start with ConnectionConfigInfo.from_dict and the schema definitions for IcebergRestConnectionConfigInfo, HadoopConnectionConfigInfo, and HiveConnectionConfigInfo, comparing their allOf layout with AwsStorageConfigInfo. Regenerate the Python client and add focused discriminator-deserialization coverage for remoteCatalogName and warehouse; done when setup export preserves those values and the required client, unit, lint, build, and repository checks pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- openapi, python
- Domain
- api, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100