Default FreeSerif font not found by xelatex due to specifying Extension

Describe the bug

When specifying the latex engine as latex_engine = 'xelatex', the generated .tex file has the following font configuration:

\setmainfont{FreeSerif}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Italic,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldItalic
]
\setsansfont{FreeSans}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Oblique,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldOblique,
]
\setmonofont{FreeMono}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Oblique,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldOblique,
]

I was puzzled to encounter the error:

! Package fontspec Error: The font "FreeSerif" cannot be found.

Although the font-gnu-freefont package was installed on my system and visible via fc-list:

fc-list | grep 'FreeSerif.*otf'
/gnu/store/pdj0hwrri2riw5av1k0dsvaxfda891yw-profile/share/fonts/opentype/FreeSerifBold.otf: FreeSerif:style=Bold,получерен,negreta,tučné,fed,Fett,Negrita,Lihavoitu,Gras,Félkövér,Grassetto,Vet,Halvfet,Pogrubiony,Negrito,gros,Обычный,Fet,ตัวหนา,Kalın,huruf tebal,жирний,Polkrepko,treknraksts,pusjuodis,ضخیم,đậm,Lodia,धृष्ट
/gnu/store/pdj0hwrri2riw5av1k0dsvaxfda891yw-profile/share/fonts/opentype/FreeSerif.otf: FreeSerif:style=Regular,нормален,normal,obyčejné,Mittel,µεσαία,Normaali,Normál,Normale,Gemiddeld,odmiana zwykła,Обычный,Normálne,ปกติ,menengah,прямій,Navadno,vidējs,normalusis,عادی,vừa,Arrunta,सामान्य
/gnu/store/pdj0hwrri2riw5av1k0dsvaxfda891yw-profile/share/fonts/opentype/FreeSerifItalic.otf: FreeSerif:style=Italic,курсивен,cursiva,kurzíva,kursiv,Λειψίας,Kursivoitu,Italique,Dőlt,Corsivo,Cursief,kursywa,Itálico,cursiv,Курсив,ตัวเอียง,İtalik,kursif,Ležeče,kursīvs,kursivas,nghiêng,Etzana,तिरछा
/gnu/store/pdj0hwrri2riw5av1k0dsvaxfda891yw-profile/share/fonts/opentype/FreeSerifBoldItalic.otf: FreeSerif:style=Bold Italic,получерен курсивен,negreta cursiva,tučné kurzíva,fed kursiv,Fett-Kursiv,Negrita Cursiva,Lihavoitu Kursivoi,Gras Italique,Félkövér dőlt,Grassetto Corsivo,Vet Cursief,Halvfet Kursiv,Pogrubiona kursywa,Negrito Itálico,gros cursiv,Обычный Курсив,Tučná kurzíva,Fet Kursiv,ตัวเอียงหนา,Kalın İtalik,huruf tebal kursif,жирний курсив,Polkrepko Pežeče,treknais kursīvs,pusjuodis kursyvas,nghiêng đậm,Lodi etzana,धृष्ट-

After asking in #latex on libera.chat (IRC), it was diagnosed to be because when specifying Extension, the font is looked by its file name in the TEXMF tree by kpathsea which fails here because the font is not installed to the TEXMF tree. Removing the Extension lines above, the font is correctly found by xelatex through fontconfig.

I'm not sure what the best course of action is here. Could dropping the Extension line work for everyone (including those using the texlive package for the GNU FreeFont rather than as a "regular" font package) ?

How to Reproduce

It depends of your distribution, but on Debian you'd need to have the fonts-freefont-otf package installed and not have the texlive-fonts-extra-links package installed to trigger the issue.

$ git clone https://github.com/ipython/ipython
$ cd ipython
$ cd docs
$ echo "latex_engine = 'xelatex'" >> source/conf.py
$ pip install -r requirements.txt
$ make pdf

Expected behavior

The fonts should be found by xelatex.

Your project

N/A

Screenshots

No response

OS

GNU/Linux

Python version

3.9.9

Sphinx version

4.2.0

Sphinx extensions

No response

Extra tools

No response

Additional context

No response

Reproduce the issue from docs/source/conf.py by setting latex_engine to xelatex and running make pdf. Inspect the generated .tex font configuration and verify that PDF generation finds FreeSerif with both regular and TeX Live font installations; done means the documented reproduction no longer produces the fontspec error.