jupyter / jupyter/nbconvert

Performance: skip repeated validator runs between preprocessors

Open
#1,663 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.9k
Forks
613
Avg merge
2h 11m
Merged PRs (30d)
1

Description

Hello! I've been digging into some slow nbconvert runs with flamegraphs, and I noticed a lot of time (~60% of the overall run) being spent on validation. That's because it runs validation between every preprocessor: https://github.com/jupyter/nbconvert/blob/main/nbconvert/exporters/exporter.py#L320. The HTML exporter has about ten preprocessors, which means we do a lot of revalidation.

For my use case (building a nbconvert rendering server), I'm not interested in which preprocessor failed validation, so I'd prefer to only run validation at the beginning and end of `_preprocess`. Patching this out locally speeds up conversion pretty significantly (by 1.5 seconds on a ~400kb notebook).

I'd like to add a env var or config option to change the behaviour of `_preprocess` to what I described above.

I can write my own exporter class that overrides `_preprocess`, but as the method changes over time it might be tough to keep it up to date.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.