[Task] - Reduce git repository sizes - cleanup PR docs previews from repos

Description

Goal: Reduce the size of our git repositories by cleaning up PR docs previews in projects. Setting up automatic cleanup and reducing redundant and duplicate files.

The size of our git repositories is growing due to the accumulation of PR docs previews. This increases the time it takes to clone our projects, slowing down development and CI. And increases the size of our containers (since we currently publish cloned repos inside the containers)

Example: Transformers4Rec. The size of the code in the main development branch is 3.5MB. The size of the gh-pages branch is 939MB. About two thirds of this is PR docs previews in the review directory

The preview which is setup to build on PRs for most of our projects with the docs-preview-pr.yaml workflow.

When this job runs it saves the docs html build directory along with all static files as a git commit on the gh-pages branch of our repo to a directory review/pr-XXX.

Proposal of this task:

• Find a way to cleanup PR docs previews in gh-pages automatically
  • Potentially by impementing a step in docs-preview-pr.yaml workflow
  • Making sure to exclude .doctrees from builds which contributes approximately half the size of the static files data in Transformers4Rec for example.
    • Perhaps in a .gitignore file on the gh-pages branch
• A follow-up task (perhaps needs it's own issue) could be to try to avoid duplicate files in the multi-version builds.
  • As we increase the number of versioned releases with dedicated docs. We could further reduce the size of the gh-pages branch by putting large static shared assests (e.g. fonts) into one place and referring to them across each release

https://github.com/NVIDIA-Merlin/Merlin/issues/730#tasklist-block-574ed5ca-c8c1-469f-84ce-1d547b3375f1

Start by inspecting the docs-preview-pr.yaml workflow and the gh-pages review/pr-XXX layout in the Transformers4Rec example. Determine how previews are currently committed, then verify that automatic cleanup and exclusion of .doctrees reduce accumulated preview data without breaking active previews.