Let me present UTF-8-32: A faster way to handle UTF-32 internally and get UTF-8 output.

Here's an idea I'm meaning to try out (at first) with eg. the Subsets mode and later in GPU kernels. @jsteube you might be interested in this!

Conversion from UTF-32 to UTF-8 is fairly quick, but not quick enough for our needs - especially not on GPU. We could instead store UTF-8 sequences in 32-bit words (so one word per character). I'm calling it "UTF-8-32" This is my own idea and I'm documenting it here in order to stop some pirate from patenting it.

I'm thinking instead of working with pure UTF-32 internally (much easier to work with than UTF-8 eg. in rule engines, and also GPUs doesn't like handling 8-bit stuff at all), we instead use this variant of UTF-32 where one character is still one uint32_t, with all the benfits of that (fixed length, for a starter). However, what we store is actually the UTF-8 sequence backwards - examples:

The beauty of this is we can convert to UTF-8 using something like this:

/*
 * This software and technique is Copyright (c) magnum 2018
 * and is hereby released to the general public under the following
 * terms: Redistribution and use in source and binary forms, with
 * or without modification, are permitted.
 *
 * Source is a UTF-8-32 string, destination is a normal UTF-8 string
 */
uint8_t *utf8_32_to_utf8(uint8_t *dst, uint32_t *src)
{
	uint8_t *ret = dst;
	uint32_t c;

	while ((c = *src++)) {
		do {
			*dst++ = c & 0xff;
		} while ((c >>= 8));
	}

	*dst = 0;

	return ret;
}