apache / apache/arrow-rs

Arrow IPC integration test "generated_map_non_canonical" failure

Open
#3,460 7 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
3.6k
Forks
1.3k
Avg merge
2d 14h
Merged PRs (30d)
167

Description

**Describe the bug**
There is some problem reading the Arrow Map files from the IPC format

in the reader tests: https://github.com/apache/arrow-rs/blob/81abc1a942cd13a92231f4a828077ad60fdabe36/arrow-integration-testing/tests/ipc_reader.rs#L120-L122

and the writer tests: https://github.com/apache/arrow-rs/blob/81abc1a942cd13a92231f4a828077ad60fdabe36/arrow-integration-testing/tests/ipc_writer.rs#L73-L74

**To Reproduce**
Uncomment one the lines referenced above and the test will fail like the following (looks like the names of the fields are different -- "entries" vs "some_entries")

```
Verifying /Users/alamb/Software/arrow-rs2/arrow/../testing/data/arrow-ipc-stream/integration/1.0.0-littleendian/generated_map_non_canonical.stream
verifying content
thread 'read_1_0_0_littleendian' panicked at 'assertion failed: `(left == right)`
left: `Field { name: "map_other_names", data_type: Map(Field { name: "some_entries", data_type: Struct([Field { name: "some_key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "some_value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }`,
right: `Field { name: "map_other_names", data_type: Map(Field { name: "entries", data_type: Struct([Field { name: "key", data_type: Utf8, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, Field { name: "value", data_type: Int32, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }]), nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} }, false), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }`: Arrow fields not the same', arrow-integration-test/src/lib.rs:253:17
stack backtrace:
0: rust_begin_unwind
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
2: core::panicking::assert_failed_inner
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:240:23
3: core::panicking::assert_failed
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:203:5
4: arrow_integration_test::ArrowJsonField::equals_field
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:253:17
5: arrow_integration_test::ArrowJsonSchema::equals_schema
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:217:17
6: arrow_integration_test::ArrowJson::equals_reader
at /Users/alamb/Software/arrow-rs2/arrow-integration-test/src/lib.rs:166:13
7: ipc_reader::verify_arrow_stream
at ./tests/ipc_reader.rs:218:17
8: ipc_reader::read_1_0_0_littleendian::{{closure}}
at ./tests/ipc_reader.rs:140:9
9: as core::iter::traits::iterator::Iterator>::for_each
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/iter/macros.rs:211:21
10: ipc_reader::read_1_0_0_littleendian
at ./tests/ipc_reader.rs:138:5
11: ipc_reader::read_1_0_0_littleendian::{{closure}}
at ./tests/ipc_reader.rs:109:1
12: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
13: core::ops::function::FnOnce::call_once
at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
```
**Expected behavior**

**Additional context**
Found on https://github.com/apache/arrow-rs/pull/3427

Contributor guide

Open the contributing guide

Research direction

Start by uncommenting the referenced cases in arrow-integration-testing/tests/ipc_reader.rs and ipc_writer.rs, then run the failing integration tests. Read arrow-integration-test/src/lib.rs around ArrowJsonField::equals_field and compare the generated_map_non_canonical schema fields. Done means the reader and writer integration tests pass for this Map file.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.