apache / apache/beam

Beam transforms reorder fields

Open
#18,916 0 comments 0 reactions 0 assignees View on GitHub
core java P3 sub-task
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

Currently transforms such as Select, DropFields, RenameFields, and AddFields can create schemas with unexpected order. The problem is that FieldAccessDescriptor stores top-level fields and nested fields separately, so there's no way to tell the relative order between them. To fix this we should refactor FieldAccessDescriptor: instead of storing these separately it should store a single list, where each item in the list might optionally have a nested FieldAccessDescriptor.

Expected behavior from the transforms:

   DropFields: preserves order in original schema

   RenameFields: preserves order in original schema

   AddFields: adds fields in order specified. If multiple nested fields are selected, the first reference to the top field wins (e.g. adding "a.b", "c", "a.d" results in adding a before c.

  Select: Select fields in the order specified.

Imported from Jira [BEAM-7301](https://issues.apache.org/jira/browse/BEAM-7301). Original Jira may contain additional context.
Reported by: reuvenlax.
Subtask of issue #18898

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.