lincc-frameworks / lincc-frameworks/nested-pandas

NestedExtensionArray `__get_item__` loses pyarrow dtypes

Open
#321 2 comments 0 reactions 0 assignees View on GitHub
bug LSDB question
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

**Bug report**
NestedExtensionArray `__get_item__` appears to not preserve pyarrow dtypes for doubly nested datasets, where the deepest layer still has the pyarrow dtypes, but any middle layers revert to numpy dtypes, see the failed test in #320 for a reproducer. The failing test is specifically designed to catch this, but it was passing previously because prior to #320, generate_data wasn't fully using pyarrow dtypes. In #320, I started trying to fix this, but it had cascading changes that were causing more and more tests to fail, so instead I just want to capture this as an issue for now.

**Before submitting**
Please check the following:

- [x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the failed test referenced in #320 and the implementation of NestedExtensionArray.__get_item__. Trace dtype handling for doubly nested datasets, then confirm that pyarrow dtypes are preserved through the middle layers and that the failing test passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.