dotnet / dotnet/docfx

[Bug] Exponential growth of template processing time for large repositories due to shared RawModels

Open
#10,506 0 comments 0 reactions 0 assignees View on GitHub
template
Dominant language
C#
Stars
4.4k
Forks
890
Avg merge
2h 11m
Merged PRs (30d)
10

Description

**Describe the bug**
When using the [templating system](https://dotnet.github.io/docfx/tutorial/intro_template.html) it is often desirable to set the `isShared` flag in the function `exports.getOptions` to `true` in order to access information from other objects to customize the API template. However, this leads to growing of the `RawModel` size, which becomes an issue especially for large repositories and leads to exponential growth of the `applying template to X models` part of the `docfx` build process.

**To Reproduce**
Steps to reproduce the behavior:
1. Choose a large repository (few hundred - thousand API objects)
2. Create a template folder with a file called 'ManagedReference.overwrite.js`.
```javascript
exports.getOptions = function (model) {
return {
isShared: true
};
}
```
3. Add template folder to `docfx.json` `template` property.
4. Run
```powershell
dotnet docfx --exportRawModel
```
5. On a repository that produces ~1000 models, this takes 30 minutes on my computer and generates ~1000 individual `RawModel` files that are each ~70Mb in size and >1.8 million lines. Even if the `RawModel` files are not being exported, template processing is very slow, so I think the problem is just that the input data model is just very large which affects the `exports.transform` function for every item.

**Expected Behaviour**
While I think it is expected behaviour, I think it could be optimized.

**Desired Solution**
Is there a way to control what is being shared, such that the `RawModel` size is not so large? I see two possible options:
1) Filtering which `RawModels` to share.
2) Filtering which items in a `RawModel` are being shared.

This might already be doable with `exports.getOptions`, but I am not sure how, so if someone can point me to an example that would be great!

**Context (please complete the following information):**
- Browser: Chrome
- Form factor: Desktop
- Docfx version: 2.78.2
- Template: default, custom.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.