Psych.dump raises Encoding::CompatibilityError for UTF-16/UTF-32 strings

Summary

Psych.dump raises Encoding::CompatibilityError for any string in a non-ASCII-compatible encoding (UTF-16LE/BE, UTF-32LE/BE). An empty string is enough to trigger it.

This is asymmetric with the load side: Psych.load explicitly supports UTF-16 input and has tests for it (test_transcode_utf16le / test_transcode_utf16be in test/psych/test_encoding.rb), but dumping a UTF-16 string crashes.

require "psych"

Psych.load("--- こんにちは！".encode("UTF-16LE"))  # => "こんにちは！"  (works)
Psych.dump("".encode("UTF-16LE"))                  # => raises

Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp with UTF-16LE string)
    lib/psych/visitors/yaml_tree.rb:303:in 'String#match?'
    lib/psych/visitors/yaml_tree.rb:303:in 'Psych::Visitors::YAMLTree#visit_String'

Affects every dump entry point: Psych.dump, YAML.dump, and to_yaml on a String, or on any Hash/Array/nested structure containing one — including when the string is used as a hash key.

Cause

visit_String matches the string against US-ASCII regexp literals:

• yaml_tree.rb:303 — o.match?(/\n(?!\Z)/)
• yaml_tree.rb:313 — o.match?(/^[^[:word:]][^"]*$/)
• yaml_tree.rb:316 — /\A0[0-7]*[89]/.match?(o)

A string whose encoding is not ASCII-compatible cannot be matched against an ASCII regexp at all, so the first of these raises regardless of the string's content.

The guard above them only covers ASCII_8BIT:

def binary? string
  string.encoding == Encoding::ASCII_8BIT && !string.ascii_only?
end

so UTF-16/32 falls straight through.

Which encodings are affected

The crashing set maps exactly onto Encoding#ascii_compatible?:

Every other non-UTF-8 encoding transcodes and round-trips correctly, which suggests the UTF-16/32 family is simply an unhandled case rather than a deliberate restriction.

Expected

A UTF-16/32 string should dump like the equivalent UTF-8 string does — it should not be tagged !binary, and it should round-trip.

Environment

Reproduced on psych 5.4.0 (Ruby 4.0.4) and on master 79be592f1dbe3c6eb36d4e1015b8379207158a75 (5.5.0), same file and line on both.

I have a patch and will open a PR referencing this issue.

Inspect Psych::Visitors::YAMLTree#visit_String around lines 303, 313, and 316 in lib/psych/visitors/yaml_tree.rb. Start with the UTF-16 encoding coverage in test/psych/test_encoding.rb, including test_transcode_utf16le and test_transcode_utf16be, then verify that UTF-16/32 strings dump without an encoding error and round-trip like equivalent UTF-8 strings.