lincc-frameworks / lincc-frameworks/nested-pandas
NestedExtensionArray `__get_item__` loses pyarrow dtypes
- Dominant language
- Python
- Stars
- 26
- Forks
- 8
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 9
Description
**Bug report**
NestedExtensionArray `__get_item__` appears to not preserve pyarrow dtypes for doubly nested datasets, where the deepest layer still has the pyarrow dtypes, but any middle layers revert to numpy dtypes, see the failed test in #320 for a reproducer. The failing test is specifically designed to catch this, but it was passing previously because prior to #320, generate_data wasn't fully using pyarrow dtypes. In #320, I started trying to fix this, but it had cascading changes that were causing more and more tests to fail, so instead I just want to capture this as an issue for now.
**Before submitting**
Please check the following:
- [x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the failed test referenced in #320 and the implementation of NestedExtensionArray.__get_item__. Trace dtype handling for doubly nested datasets, then confirm that pyarrow dtypes are preserved through the middle layers and that the failing test passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100