Hyperref applies in bookmarks EN-DASH and EM-DASH TeX ligatures but not double quotes ones, and extra twist with xelatex

With pdflatex

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[unicode]{hyperref}

\begin{document}

\section{A-section--with---hyphens}

\section{A `section' with ``quotes''}

\end{document}

produces this in .out file

\BOOKMARK [1][-]{section.1}{\376\377\000A\000-\000s\000e\000c\000t\000i\000o\000n\040\023\000w\000i\000t\000h\040\024\000h\000y\000p\000h\000e\000n\000s}{}% 1
\BOOKMARK [1][-]{section.2}{\376\377\000A\000\040\000`\000s\000e\000c\000t\000i\000o\000n\000'\000\040\000w\000i\000t\000h\000\040\000`\000`\000q\000u\000o\000t\000e\000s\000'\000'}{}% 2

and one sees bookmark will contain en-dash and em-dash.

I don't have pdftk and did not try to export the bookmark. The closing two double quotes look in this screenshot as a " but maybe this is just '' as the .out file seems to indicate.

Thus, the issue here is one of coherence between handling of hyphens and of quote characters. Usually people do not disable TeX ligatures. But the hyperref handling of hyphens looks so far ok because if the input did contain -{}- then the bookmark would contain separated hyphens, matching the typeset text.

edit: also << and >> end up as is in bookmarks, thus not matching the typeset text (T1 encoding) and behaving in contradiction with -- and ---.

But with xelatex, a new twist arises. It is useful to disable TeX ligatures, especially for documents not manually written but converted from other sources and for this context if the author uses --- it was with the intent to see it as is in ouput, else a Unicode codepoint would have been used in input. Consider this

\documentclass{article}
\usepackage{fontspec}
\defaultfontfeatures[\rmfamily,\sffamily]{}% turn off TeX ligatures

\setmainfont{FreeSerif}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Italic,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldItalic
]
\setsansfont{FreeSans}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Oblique,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldOblique,
]
\setmonofont{FreeMono}[
  Extension      = .otf,
  UprightFont    = *,
  ItalicFont     = *Oblique,
  BoldFont       = *Bold,
  BoldItalicFont = *BoldOblique,
]

\usepackage[unicode]{hyperref}

\begin{document}
\section{A-section--with---hyphens}

\section{A `section' with ``quotes''}

\end{document}
% Local variables:
% TeX-engine: xetex
% End:

then we get again

\BOOKMARK [1][-]{section.1}{\376\377\000A\000-\000s\000e\000c\000t\000i\000o\000n\040\023\000w\000i\000t\000h\040\024\000h\000y\000p\000h\000e\000n\000s}{}% 1
\BOOKMARK [1][-]{section.2}{\376\377\000A\000\040\000`\000s\000e\000c\000t\000i\000o\000n\000'\000\040\000w\000i\000t\000h\000\040\000`\000`\000q\000u\000o\000t\000e\000s\000'\000'}{}% 2

thus

which is disturbing regarding dashes but better regarding quotes as now the typeset text matches the bookmarks.

edit: as << and >> end up as is in bookmarks, they do match the typeset text in this case, like quotes. But contrarily to hyphens.

Start with the pdflatex and xelatex minimal examples in the issue and compare their generated .out bookmark entries with the typeset section text. Trace hyperref's bookmark handling for --, ---, quotes, and angle brackets under both engines and encoding settings. Done means bookmark text consistently matches the rendered text without breaking the documented ligature behavior.