Cross-reference take most of time when building large package with a probable bug in make_xref

Describe the bug

Building large packages takes a lot of time, most of which is cost on resolving cross-references.

When building a package of 70MB and 3399 files (azure-mgmt-containerservice), it cost 3615 seconds and 2806 seconds are cost on resolving cross reference.

"Fuzzy" searching mode of PythonDomain.find_obj method is iterating all loaded objects in the memory, when the package has a lot of objects and xref to resolve, the build process is very slow.

The searchmode parameter is designed to control whether using Fuzzy mode to resolve cross references, but I think it's not working under most cases (e.g. if I cross-reference the built-in :class: str, Sphinx will use fuzzy mode to match it).

I think the problem is in PyXrefMixin.make_xref, attribute 'refspecific' is set to True, and won't be changed whatever the reftarget string is. Is it the expected behavior? Update the node attributes according to "refspecific" in return value back seems more reasonable after parsing reftarget.

And can we have a switch in config files to only find exact match and don't fuzzy search when building a package?

Screenshot:
Full iterating for fuzzy mode:

Seems should update the node attributes "refspecific" after parse_reftarget.

How to Reproduce

Reproduce python script:

from sphinx.application import Sphinx

app = Sphinx(
        srcdir= './rst',
        confdir='./conf',
        outdir= './output',
        buildername='html',
        doctreedir='./doctrees'
    )
app.build()

Command to run cprofile:

 python -m cProfile -o "stats-compare" -s 'cumulative' .\run-sphinx-with-cprofile.py

RSTs are generated by apidoc:

sphinx-apidoc --module-first --no-headings --no-toc --implicit-namespaces -o "" ""

conf.py:

# -*- coding: utf-8 -*-

source_suffix = '.rst'
master_doc = 'index'
project = u''
copyright = u''
author = u''
version = '0.1'
release = '0.1'
language = None
exclude_patterns = ['_build']
pygments_style = 'sphinx'
todo_include_todos = False
html_theme = 'bizstyle'
html_static_path = ['_static']
htmlhelp_basename = 'Example Document'
extensions = [
    'sphinx.ext.autodoc',
    'sphinx.ext.napoleon'
]

intersphinx_mapping = {'python': ('https://docs.python.org/3.6', None)}
# Make Google-style and Numpy-style Example work.
napoleon_use_admonition_for_examples = True

exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
autodoc_mock_imports = ['google']
autoclass_content = 'both'
autodoc_default_options = {
    'inherited-members': True
}

Environment Information

Platform:              win32; (Windows-10-10.0.22621-SP0)
Python version:        3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.19
Jinja2 version:        3.0.1
Pygments version:      2.14.0

Sphinx extensions

['sphinx.ext.autodoc']

Additional context

No response

Start with sphinx/domains/python.py, especially PyXrefMixin.make_xref and PythonDomain.find_obj, then run the supplied Sphinx build and cProfile reproduction against the large package scenario. Trace how refspecific and searchmode affect fuzzy lookup; done means the cross-reference behavior and configuration request are resolved without the reported large-build slowdown.