Cref links not working when running docfx from workflow

**Operating System**: (`Windows` or `Linux` or `MacOS`)
Windows
**DocFX Version Used**: 2.56.4

**Template used**: (`default` or `statictoc` or contain custom template)

Default

**Steps to Reproduce**:

DocFx cannot find crefs on other public repositories on DocFx run without running Visual Studio build.

I added this flag to the docfx.json.
```
"xref": [
"https://relativitydev.github.io/relativity.testing.framework/xrefmap.yml"
]
```

When DocFx is run from a clean repository with no cache, the crefs are unable to load. The xrefs seem to be loaded during the build step but are needed during the metadata to build everything up.

1. Have a repository with external crefs in the documentation
2. Run git clean -fdx to make sure cache is cleared
3. Running DocFx once leaves this warning for external crefs: invalid cref value

Only when DocFx is run after Visual Studio build it works.

**Expected Behavior**:
This is what should happen, and it happens when DocFx is run a second time. This happens when Visual studio

**Actual Behavior**:
This is what happens when DocFx is run once with external crefs when the cache is clear.