AnswerDotAI / AnswerDotAI/nbdev

Error building _modidx.py

Aperta
#1,153 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
bug
Lingua principale
Jupyter Notebook
Stelle
5.3k
Fork
513
Merge medio
2g 30m
PR unite (30g)
8

Descrizione

If the library folder contains modules extraneous to **nb_dev**, building `_modidx.py` will fail in some circumstances. In particular, parsing files generated by **VSCode** *export to python script* (`@command:jupyter.exportAsPythonScript`) and probably those generated by [Jupytext](https://jupytext.readthedocs.io/en/latest/#)

## Explanation
Looking at the code for building `_modidx.py`, it seems the intent is to only parse *nbdev* controlled modules:
```python
def _iter_py_cells(p):
"Yield cells from an exported Python file."
p = Path(p)
cells = p.read_text().split("\n# %% ")
for cell in cells[1:]:
top,code = cell.split('\n', 1)
nb,idx = top.split()
nb_path = None if nb=='auto' else (p.parent/nb).resolve() # NB paths are stored relative to .py file
if code.endswith('\n'): code=code[:-1]
yield AttrDict(nb=nb, idx=int(idx), code=code, nb_path=nb_path, py_path=p.resolve())
```
But the parsing of the cell header can easily fail if it encounters cell tags not generated by *nbdev*, i.e., `# %% [markdown]`, as seen below.

## Details
### Recreate the issue
- clone [civvic/nbdev-mixed-lib](https://github.com/civvic/nbdev-mixed-lib)
- run `nbdev_export`

```bash
❯ nbdev_export
Traceback (most recent call last):
File "/Users/vic/mambaforge/envs/TBOs/bin/nbdev_export", line 33, in
sys.exit(load_entry_point('nbdev', 'console_scripts', 'nbdev_export')())
File "/Users/vic/dev/repo/source/fastai/fastcore/fastcore/script.py", line 119, in _f
return tfunc(**merge(args, args_from_prog(func, xtra)))
File "/Users/vic/dev/repo/source/fastai/nbdev/nbdev/doclinks.py", line 137, in nbdev_export
_build_modidx()
File "/Users/vic/dev/repo/source/fastai/nbdev/nbdev/doclinks.py", line 99, in _build_modidx
res['syms'].update(_get_modidx((dest.parent/file).resolve(), code_root, nbs_path=nbs_path))
File "/Users/vic/dev/repo/source/fastai/nbdev/nbdev/doclinks.py", line 70, in _get_modidx
for cell in _iter_py_cells(py_path):
File "/Users/vic/dev/repo/source/fastai/nbdev/nbdev/doclinks.py", line 53, in _iter_py_cells
nb,idx = top.split()
ValueError: not enough values to unpack (expected 2, got 1)
```
or simply add a file with these content to lib folder:
```python
# %%
import sys

# %% [markdown]
# `Stage` corresponds to one node.

# %%
print(sys.path)
```
### Workaround
I've quickly patched `maker.ipynb/py` and `doclinks.ipynb/py` to filter out files without headers generated by nbdev (`# AUTOGENERATED! DO NOT EDIT! File to edit:...`).

My understanding is `_modidx.py` is mainly used for doc generation, bu I'm not sure, so probably it would be a good idea to strengthen the parser to extract symbols from modules other than those controlled by `nb_dev`. But in the meantime, I want to continue exploring the new `nbdev` (I've used v1 a lot in mixed environment with very few problems)

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia in nbdev/doclinks.py, in _iter_py_cells e nel percorso _build_modidx mostrato nel traceback; confronta gli header generati da nbdev con le celle in stile VSCode e Jupytext presenti nell’issue. Esegui nbdev_export sulla libreria mista o sull’esempio fornito. Il lavoro è completato quando i file Python estranei non causano più un ValueError durante la creazione di _modidx.py.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
jupyter-notebook, python
Ambito
documentation, tooling
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.