emilydolson / emilydolson/phylotrackpy

Mysterious interaction with Jupyter/IPython causes deserialization to hang

Open
#91 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
12
Forks
2
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

This bug is deeply, deeply cursed. :mage:

Somehow, the Jupyter environment interacts with phylotrackpy to cause deserialization to hang.
However, once you interrupt and run again it works fine. Does not occur when running interactively in shell.

To workaround, it is necessary to use `multiprocess` to perform phylotrackpy operations in a forked process.
Here's some example code for the next time this issue is encountered.
```python
import multiprocessing

records = []
for replicate, tree_df in tqdm(df.groupby("replicate")):
tree_df = tree_df.reset_index(drop=True)
attrs = {
col: mit.one(tree_df[col].unique())
for col in tree_df.columns
if len(tree_df[col].unique()) == 1 and col not in ["dataSource"]
}

def calc_mean_evolutionary_distinctiveness(tree_df):
tree = apc.RosettaTree(tree_df).as_phylotrack
return tree.get_mean_evolutionary_distinctiveness(
tree_df["origin_time"].max()
)

def get_mean_evolutionary_distinctiveness(tree_df):
with multiprocessing.Pool(1) as pool:
result = pool.map(calc_mean_evolutionary_distinctiveness, [tree_df])
return result[0]

records.append(
{
**attrs,
"replicate": replicate,
"metric": "sum branch lengths",
"value": alifestd_sum_origin_time_deltas_asexual(tree_df),
},
)
records.append(
{
**attrs,
"replicate": replicate,
"metric": "mean evolutionary distinctiveness",
"value":get_mean_evolutionary_distinctiveness(tree_df),
},
)

dfmetrics = pd.DataFrame.from_records(records)
```

**To Reproduce**
Steps to reproduce the behavior:

```
mv reproduce.ipynb{.json,}
jupyter nbconvert --execute --inplace reproduce.ipynb
```
Also occurs in interactive sessions.

[out.csv](https://github.com/emilydolson/phylotrackpy/files/14968508/out.csv)
[reproduce.ipynb.json](https://github.com/emilydolson/phylotrackpy/files/14968522/reproduce.ipynb.json)
* had to make JSON for GH issues upload compatibility

**Expected behavior**
A clear and concise description of what you expected to happen.

Notebooks should not hang when using phylotrackpy.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Computational environment (please complete the following information):**
- OS: Fedora
- Python version 3.10.4

```
Last updated: 2024-04-13T15:52:06.351382-04:00

Python implementation: CPython
Python version : 3.10.14
IPython version : 8.22.1

Compiler : GCC 13.2.1 20240316 (Red Hat 13.2.1-7)
OS : Linux
Release : 6.8.4-200.fc39.x86_64
Machine : x86_64
Processor :
CPU cores : 8
Architecture: 64bit

Git hash: 008cdf2d71f41e37c3f6b4539121ff39fb53310e

Git branch: debug2

phylotrackpy : 0.2.0
seaborn : 0.13.2
numpy : 1.23.5
alifedata_phyloinformatics_convert: 0.16.2
more_itertools : 9.1.0
pandas : 1.5.2
joblib : 1.3.2
teeplot : 1.0.1
keyname : 0.5.2

Watermark: 2.4.3
```

**Additional context**
Mostly adding this to the issue tracker so we can +1 it if encountered again.

Contributor guide

Open the contributing guide

Research direction

Start by running the attached reproduce.ipynb.json with `jupyter nbconvert --execute --inplace reproduce.ipynb` under the listed Python and Jupyter versions, then compare the behavior with an interactive shell session. Trace the deserialization operation that hangs; done means the notebook completes without interruption and produces its expected output.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter, python
Domain
backend, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.