ACP `Vec<u8>::push_utf8` for no-memcpy UTF8 in a general bytes buffer

Proposal

Problem statement

When assembling a Vec<u8> containing chunks of UTF-8 text, the current idiom is something like this:

let c: char = /* obtain a `char` from somewhere */ ;
let mut scratch = [0u8; char::MAX_LEN_UTF8];
buf.extend_from_slice(c.encode_utf8(&mut scratch).as_bytes());

Besides the verbosity, this has a drawback of emitting a memcpy.

Motivating examples or use cases

This code is safe, but slower than it could be:

fn decode_into(buf: &mut Vec<u8>, mut raw: &[u8]) {
	buf.reserve(raw.len());
	while !raw.is_empty() {
		let cap = buf.spare_capacity_mut();
		// SAFETY: `unescape` guarantees that number of bytes consumed
		// from `raw` is always >= `c.len_utf8()`, so the initial reservation is
		// enough for the entire loop.
		unsafe { core::hint::assert_unchecked(cap.len() >= char::MAX_LEN_UTF8) };
		let mut utf8 = [0u8; char::MAX_LEN_UTF8];
		let c = unescape(&mut raw);
		buf.extend_from_slice(c.encode_utf8(&mut utf8).as_bytes());
	}
}

This code is fast, but has unclear safety:

fn decode_into(buf: &mut Vec<u8>, mut raw: &[u8]) {
	buf.reserve(raw.len());
	while !raw.is_empty() {
		let cap = buf.spare_capacity_mut();
		// SAFETY: as above
		unsafe { core::hint::assert_unchecked(cap.len() >= char::MAX_LEN_UTF8) };
		let mut utf8 = [0u8; char::MAX_LEN_UTF8];
		let c = unescape(&mut raw);
		// SAFETY: ????
		let utf8_len = unsafe { c.encode_utf8(transmute(cap)) }.len();
		unsafe { buf.set_len(buf.len() + utf8_len); }
	}
}

It's transmuting a possibly uninitialized &mut [MaybeUninit<u8>] to a &mut [u8], and the only reason it's not obviously mad is that a human knows that char::encode_utf8() can be expected not to read from its scratch buffer.

Solution sketch

impl Vec<u8> {
	pub fn push_utf8(&mut self, c: char) {
		// does the same as fast version above, but behind a safe API that can avoid
		// undefined transmuting.
	}
}

used like:

let c = unescape(&mut raw);
buf.push_utf8(c);

Alternatives

Add a new function to char that accepts an uninitialized buffer of the expected size:

impl char {
	// Same as today, but allowing `dst` to be uninitialized
	pub fn encode_utf8_to(self, dst: &mut [MaybeUninit<u8>]) -> &mut str;

	// Adding a size constraint with the recent `MAX_LEN_UTF8`? This will
	// statically guarantee no panic, but might be more annoying at the call site.
	pub fn encode_utf8_to(self, dst: &mut [MaybeUninit<u8>; char::MAX_LEN_UTF8]) -> &mut str;
}

Links and related work

What happens now?

This issue contains an API change proposal (or ACP) and is part of the libs-api team feature lifecycle. Once this issue is filed, the libs-api team will review open proposals as capability becomes available. Current response times do not have a clear estimate, but may be up to several months.

Possible responses

The libs team may respond in various different ways. First, the team will consider the problem (this doesn't require any concrete solution or alternatives to have been proposed):

• We think this problem seems worth solving, and the standard library might be the right place to solve it.
• We think that this probably doesn't belong in the standard library.

Second, if there's a concrete solution:

• We think this specific solution looks roughly right, approved, you or someone else should implement this. (Further review will still happen on the subsequent implementation PR.)
• We're not sure this is the right solution, and the alternatives or other materials don't give us enough information to be sure about that. Here are some questions we have that aren't answered, or rough ideas about alternatives we'd want to see discussed.

Start with the problem statement, solution sketch, and alternatives in this ACP; the issue names no implementation files or tests. Compare the proposed Vec::push_utf8 API with the char::encode_utf8_to alternatives and review the libs-api feature lifecycle. Done means the API direction is resolved and an implementation scope is agreed.