astropy / astropy/astropy

Fail to index table with iloc[] after dump/load table into pickle file

Open
#11,332 12 comments 0 reactions 0 assignees View on GitHub
Bug needs-reconfirmation table
Dominant language
Python
Stars
5.3k
Forks
2.2k
Avg merge
1d 19h
Merged PRs (30d)
75

Description

### EDIT: Temporary Workaround

See https://github.com/astropy/astropy/issues/11332#issuecomment-781561392

### Description
I am saving and loading a `Table` object into a pickle file, but after loading the table cannot be indexed using the `iloc` method. I found that problem is because after loading the table doesn't the attribute `Table.primary_key`

### Steps to Reproduce
```python
from astropy.table import Table
import pickle

t = Table([(1, 2, 3, 4), (10, 1, 9, 9)], names=('a', 'b'), dtype=['i8', 'i8'])
t.add_index('a')
print(t.iloc[2]) #--> [3, 9]

with open("t_test.pkl" , "wb") as f:
pickle.dump(t, f, protocol=0)

t_ = pickle.load(open("t_test.pkl", "rb"))

print(t_.iloc[2])
```

leads to the following error:

```
-----------------------------
TypeErrorTraceback (most recent call last)
in
11 t_ = pickle.load(open("t_test.pkl", "rb"))
12
---> 13 print(t_.iloc[2])

~/.pyenv/versions/adap/lib/python3.8/site-packages/astropy/table/index.py in __getitem__(self, item)
953 else:
954 key = self.table.primary_key
--> 955 index = self.indices[key]
956 rows = index.sorted_data()[item]
957 table_slice = self.table[rows]

~/.pyenv/versions/adap/lib/python3.8/site-packages/astropy/table/index.py in __getitem__(self, item)
809 raise IndexError(f"No index found for {item}")
810
--> 811 return super().__getitem__(item)
812
813

TypeError: list indices must be integers or slices, not NoneType
```
and here when accessing the `primary_key` attribute of the original and post-pickle table:
```python
t.primary_key, t_.primary_key
# (('a',), None)
```

### System Details

macOS-10.15.7-x86_64-i386-64bit
Python 3.8.6 (default, Jan 5 2021, 15:15:33)
[Clang 12.0.0 (clang-1200.0.32.28)]
Numpy 1.19.5
astropy 4.2
Scipy 1.6.0
Matplotlib 3.3.3

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the pickle and Table.iloc example, then start in astropy/table/index.py around Index.__getitem__, where the traceback accesses primary_key. Trace how Table.primary_key is restored after unpickling and add regression coverage showing that the loaded table can use iloc and retains the expected index behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.