emilydolson / emilydolson/phylotrackpy
Mysterious interaction with Jupyter/IPython causes deserialization to hang
- Dominant language
- C++
- Stars
- 12
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
This bug is deeply, deeply cursed. :mage:
Somehow, the Jupyter environment interacts with phylotrackpy to cause deserialization to hang.
However, once you interrupt and run again it works fine. Does not occur when running interactively in shell.
To workaround, it is necessary to use `multiprocess` to perform phylotrackpy operations in a forked process.
Here's some example code for the next time this issue is encountered.
```python
import multiprocessing
records = []
for replicate, tree_df in tqdm(df.groupby("replicate")):
tree_df = tree_df.reset_index(drop=True)
attrs = {
col: mit.one(tree_df[col].unique())
for col in tree_df.columns
if len(tree_df[col].unique()) == 1 and col not in ["dataSource"]
}
def calc_mean_evolutionary_distinctiveness(tree_df):
tree = apc.RosettaTree(tree_df).as_phylotrack
return tree.get_mean_evolutionary_distinctiveness(
tree_df["origin_time"].max()
)
def get_mean_evolutionary_distinctiveness(tree_df):
with multiprocessing.Pool(1) as pool:
result = pool.map(calc_mean_evolutionary_distinctiveness, [tree_df])
return result[0]
records.append(
{
**attrs,
"replicate": replicate,
"metric": "sum branch lengths",
"value": alifestd_sum_origin_time_deltas_asexual(tree_df),
},
)
records.append(
{
**attrs,
"replicate": replicate,
"metric": "mean evolutionary distinctiveness",
"value":get_mean_evolutionary_distinctiveness(tree_df),
},
)
dfmetrics = pd.DataFrame.from_records(records)
```
**To Reproduce**
Steps to reproduce the behavior:
```
mv reproduce.ipynb{.json,}
jupyter nbconvert --execute --inplace reproduce.ipynb
```
Also occurs in interactive sessions.
[out.csv](https://github.com/emilydolson/phylotrackpy/files/14968508/out.csv)
[reproduce.ipynb.json](https://github.com/emilydolson/phylotrackpy/files/14968522/reproduce.ipynb.json)
* had to make JSON for GH issues upload compatibility
**Expected behavior**
A clear and concise description of what you expected to happen.
Notebooks should not hang when using phylotrackpy.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Computational environment (please complete the following information):**
- OS: Fedora
- Python version 3.10.4
```
Last updated: 2024-04-13T15:52:06.351382-04:00
Python implementation: CPython
Python version : 3.10.14
IPython version : 8.22.1
Compiler : GCC 13.2.1 20240316 (Red Hat 13.2.1-7)
OS : Linux
Release : 6.8.4-200.fc39.x86_64
Machine : x86_64
Processor :
CPU cores : 8
Architecture: 64bit
Git hash: 008cdf2d71f41e37c3f6b4539121ff39fb53310e
Git branch: debug2
phylotrackpy : 0.2.0
seaborn : 0.13.2
numpy : 1.23.5
alifedata_phyloinformatics_convert: 0.16.2
more_itertools : 9.1.0
pandas : 1.5.2
joblib : 1.3.2
teeplot : 1.0.1
keyname : 0.5.2
Watermark: 2.4.3
```
**Additional context**
Mostly adding this to the issue tracker so we can +1 it if encountered again.
Contributor guide
Research direction
Start by running the attached reproduce.ipynb.json with `jupyter nbconvert --execute --inplace reproduce.ipynb` under the listed Python and Jupyter versions, then compare the behavior with an interactive shell session. Trace the deserialization operation that hangs; done means the notebook completes without interruption and produces its expected output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter, python
- Domain
- backend, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100