Fail to index table with iloc[] after dump/load table into pickle file
- Dominant language
- Python
- Stars
- 5.3k
- Forks
- 2.2k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 75
Description
### EDIT: Temporary Workaround
See https://github.com/astropy/astropy/issues/11332#issuecomment-781561392
### Description
I am saving and loading a `Table` object into a pickle file, but after loading the table cannot be indexed using the `iloc` method. I found that problem is because after loading the table doesn't the attribute `Table.primary_key`
### Steps to Reproduce
```python
from astropy.table import Table
import pickle
t = Table([(1, 2, 3, 4), (10, 1, 9, 9)], names=('a', 'b'), dtype=['i8', 'i8'])
t.add_index('a')
print(t.iloc[2]) #--> [3, 9]
with open("t_test.pkl" , "wb") as f:
pickle.dump(t, f, protocol=0)
t_ = pickle.load(open("t_test.pkl", "rb"))
print(t_.iloc[2])
```
leads to the following error:
```
-----------------------------
TypeErrorTraceback (most recent call last)
in
11 t_ = pickle.load(open("t_test.pkl", "rb"))
12
---> 13 print(t_.iloc[2])
~/.pyenv/versions/adap/lib/python3.8/site-packages/astropy/table/index.py in __getitem__(self, item)
953 else:
954 key = self.table.primary_key
--> 955 index = self.indices[key]
956 rows = index.sorted_data()[item]
957 table_slice = self.table[rows]
~/.pyenv/versions/adap/lib/python3.8/site-packages/astropy/table/index.py in __getitem__(self, item)
809 raise IndexError(f"No index found for {item}")
810
--> 811 return super().__getitem__(item)
812
813
TypeError: list indices must be integers or slices, not NoneType
```
and here when accessing the `primary_key` attribute of the original and post-pickle table:
```python
t.primary_key, t_.primary_key
# (('a',), None)
```
### System Details
macOS-10.15.7-x86_64-i386-64bit
Python 3.8.6 (default, Jan 5 2021, 15:15:33)
[Clang 12.0.0 (clang-1200.0.32.28)]
Numpy 1.19.5
astropy 4.2
Scipy 1.6.0
Matplotlib 3.3.3
Contributor guide
Research direction
Reproduce the failure with the pickle and Table.iloc example, then start in astropy/table/index.py around Index.__getitem__, where the traceback accesses primary_key. Trace how Table.primary_key is restored after unpickling and add regression coverage showing that the loaded table can use iloc and retains the expected index behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100