Performance: skip repeated validator runs between preprocessors
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 613
- Avg merge
- 2h 11m
- Merged PRs (30d)
- 1
Description
Hello! I've been digging into some slow nbconvert runs with flamegraphs, and I noticed a lot of time (~60% of the overall run) being spent on validation. That's because it runs validation between every preprocessor: https://github.com/jupyter/nbconvert/blob/main/nbconvert/exporters/exporter.py#L320. The HTML exporter has about ten preprocessors, which means we do a lot of revalidation.
For my use case (building a nbconvert rendering server), I'm not interested in which preprocessor failed validation, so I'd prefer to only run validation at the beginning and end of `_preprocess`. Patching this out locally speeds up conversion pretty significantly (by 1.5 seconds on a ~400kb notebook).
I'd like to add a env var or config option to change the behaviour of `_preprocess` to what I described above.
I can write my own exporter class that overrides `_preprocess`, but as the method changes over time it might be tough to keep it up to date.
Contributor guide
Assessment
This issue has not been assessed yet.