astronomy-commons / astronomy-commons/lsdb
Can't do certain queries on a DataFrame prior to LSDB import
- Dominant language
- Python
- Stars
- 55
- Forks
- 26
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 5
Description
**Bug report**
When wanting to import a `DataFrame` into LSDB, some queries have to wait until after the import or they produce an error.
```python
import json
from pathlib import Path
import pandas as pd
tns_path = Path('./tns/tns-all.json')
with tns_path.open('rb') as f_in:
tns_objs = json.load(f_in)
tns_df = pd.DataFrame(tns_objs)
tns_f = tns_df.query("name_prefix == 'AT' and type.isna()")
# This query cannot be done prior to import
tns_f = tns_f.query("source_group == 'ZTF'")
tns_db = lsdb.from_dataframe(tns_f, ra_column='ra', dec_column='declination')
tns_db
```
Error produced:
```
ValueError: Metadata mismatch found in `from_delayed`.
Partition type: `pandas.core.frame.DataFrame`
+----------------------+---------------+-----------------+
| Column | Found | Expected |
+----------------------+---------------+-----------------+
| 'Class_ADS_bibcodes' | null[pyarrow] | string[pyarrow] |
+----------------------+---------------+-----------------+
```
If this last query, `source_group == 'ZTF'`, is done *after* import into LSDB, it works fine:
```python
tns_path = Path('./tns/tns-all.json')
with tns_path.open('rb') as f_in:
tns_objs = json.load(f_in)
tns_df = pd.DataFrame(tns_objs)
tns_df.shape
tns_f = tns_df.query("name_prefix == 'AT' and type.isna()")
tns_db = lsdb.from_dataframe(tns_f, ra_column='ra', dec_column='declination')
tns_db = tns_db.query("source_group == 'ZTF'")
```
**Before submitting**
Please check the following:
- [X] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [X] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.