lincc-frameworks / lincc-frameworks/nested-pandas

Packing on an object-dtyped column may fail if items are not comparable

Open
#210 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
26
Forks
8
Avg merge
2d 2h
Merged PRs (30d)
9

Description

**Bug report**

Today @mi-dai and myself were working on packing SALT3 data into nested pandas. When we tried to run `nf.add_nested(src, on='SNID', name='lc')` it failed, because it happened that `SNID` column had object dtype and consisted of heterogeneous data: numbers and strings. A reproducible example:

```python
from nested_pandas import NestedFrame

nf = NestedFrame({"SNID": [1, 'abc']})
nf.add_nested(NestedFrame({"SNID": [1, 1, 'abc', 'abc'], "x": [1, 2, 3, 4]}), name='lc', on='SNID')
```

It failed in our code, basically because this doesn't work in pandas:
```
nd.set_index('SNID').sort_index()
```

I'm not sure if it is our problem or not. We potentially may fix this use-case with catching the `TypeError` and trying to cast `on` column values to `str`. Or we can do nothing, because we wouldn't like to change user's data and fix pandas behavior.

**Before submitting**
Please check the following:

- [x] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [x] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [x] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the reproducible NestedFrame.add_nested example in the issue and trace the path that calls set_index('SNID').sort_index(). Add a regression test using mixed numeric and string SNID values, then verify the chosen behavior handles the example without changing unrelated user data.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.