AbsaOSS / AbsaOSS/ABRiS

Preserve nested column resolution in AvroSchemaUtils.toAvroSchema

Đang mở
#375 0 bình luận 0 reaction 1 người được giao Được @kevinwallimann nhận Xem trên GitHub
Ngôn ngữ chính
Scala
Star
242
Fork
84
Merge trung bình
10 giờ 12 phút
Pull request đã merge (30 ngày)
2

Mô tả

## Summary

`AvroSchemaUtils.toAvroSchema` accepts a sequence of column names. Its current implementation builds a `StructType` from `dataFrame.schema(columnName)`. This lookup only resolves top-level schema fields.

Investigate and define the intended behavior for nested column paths such as `Seq("parent.child")`. If nested paths must be supported, preserve Catalyst column resolution when building the schema.

## Rationale

This behavior is outside the Spark 4 upgrade scope. A change can alter behavior for existing callers, so it requires separate investigation and regression coverage.

## Affected area

- `src/main/scala/za/co/absa/abris/avro/parsing/utils/AvroSchemaUtils.scala`
- Spark schema conversion tests for `AvroSchemaUtils`

## Required work

1. Identify existing callers that pass nested column paths to the `columnNames: Seq[String]` overload.
2. Define the compatibility contract for nested paths.
3. If nested paths are supported, construct the selected schema through analyzed DataFrame column expressions rather than direct top-level `StructType` field lookup.
4. Add regression tests for a nested path such as `Seq("parent.child")`.
5. Document any intentional behavior change or compatibility limitation.

## Acceptance criteria

- The nested-column behavior is explicitly defined.
- Tests cover the defined behavior on the supported Spark 4 version.
- The implementation does not unintentionally change top-level column behavior.

## Backlinks

- Source pull request: https://github.com/AbsaOSS/ABRiS/pull/370
- Source review comment: https://github.com/AbsaOSS/ABRiS/pull/370#discussion_r3898865601

Requested by: @kevinwallimann

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.