Introduce incremental `gen-api-docs` command to support watch functionality

Is your feature request related to a problem?

Currently, with the existing gen-api-docs and clean-api-docs commands at our disposal I am unable to provide a workable "watch" functionality to enable quick iterative development. Due to how these commands are designed, they do not regenerate updated MDX files after changes have been made to the source OpenAPI schema (this has already been described in https://github.com/PaloAltoNetworks/docusaurus-openapi-docs/issues/615).

So, to end up with up-to-date MDX files, we have to clean all files and regenerate them.

Our problem with this design choice is that our Docusaurus dev server seems to be stalling and going OOM after the watcher sees so many MDX files being removed and re-added by this plugin. Even adding a timeout to the watcher does not prevent this: webpackConfig.watchOptions.aggregateTimeout = 10000;. The server just hangs and ultimately crashes. This means that we have to restart the Docusaurus dev server, which means that every development iteration takes at least 30 seconds, depending on the size of your site.

As a workaround, we found that when we generate files to a temporary directory, and then use rsync -r -v --size-only to copy only the changed files from the temporary directory back to the intentional directory, this strain on Webpack does not occur and the builds are working fine. This solution however is impractical, as the info_path of the generated mdx files then contains references to our temporary build directory.

Describe the solution you'd like

I would like to ask to reconsider the idea that @shmichael proposed in https://github.com/PaloAltoNetworks/docusaurus-openapi-docs/issues/615, which is to build the full file tree & touch only the actually modified files, to reduce IO and improve watcher performance.

Additionally, to remove all deleted files, one could keep track of the generated file tree in a temporary cache file, and on the next build, remove all existing files that were not part of this tree.

I am willing to help contribute towards an implementation, however I am unsure how this fits alongside the existing workflow of the gen & clean commands. Maybe a gen-api-docs:watch command could be introduced?

This would enable a quick regeneration & development cycle.

Describe alternatives you've considered

• Build using gen-api-docs to a temporary dir, and use rsync -r -v --size-only to copy only the changed files from the temporary directory back to the intentional directory
• Use Nodemon to provide a watch function nodemon --delay 1000ms --watch <my-openapi-spec-folder> --ext 'yaml' --exec 'yarn api-docs:tmp-dir:clean && yarn gen-api-docs && yarn api-docs:tmp-dir:rsync'
• Prevent stalling the Docusaurus dev server using webpackConfig.watchOptions.aggregateTimeout = 10000;

Additional context

• https://github.com/PaloAltoNetworks/docusaurus-openapi-docs/issues/615

Thanks in advance!

Start by tracing the existing gen-api-docs and clean-api-docs commands and review the related issue #615. Define how incremental generation should preserve unchanged files, remove files absent from the generated tree, and fit with the existing gen and clean workflow. Done means source schema changes can regenerate only affected MDX files without the temporary-directory workaround or a full cleanup.