Zero-sized literal output buffers cause out-of-bounds access

Description and impact

The shared implementation behind WriteFloatHex() and WriteDoubleHex(), as well as WriteUint128(), subtracts one from the output size while truncating a formatted value. When a caller supplies size == 0, that subtraction wraps to SIZE_MAX, which leads to an invalid copy length or array index.

With an AddressSanitizer and UndefinedBehaviorSanitizer build, the float and double wrappers abort with AddressSanitizer: negative-size-param: (size=-1), while WriteUint128() aborts on an index of 18446744073709551615. Without sanitizer checks, these accesses are undefined behavior and can crash a process using libwabt.

Reproduction

Minimal library-level reproduction; run the program once with each of the three calls enabled individually:

#include "wabt/literal.h"

int main() {
  char output = 'x';
  wabt::WriteFloatHex(&output, 0, 0);
  // wabt::WriteDoubleHex(&output, 0, 0);
  // wabt::WriteUint128(&output, 0, {0, 0, 0, 0});
}

Affected surface

The affected surface is the public libwabt functions WriteFloatHex(), WriteDoubleHex(), and WriteUint128(). Their public declarations do not document a size > 0 precondition. I did not find a bundled CLI call site that supplies a zero-sized buffer; the current in-tree call sites use nonzero fixed-size buffers.

Required WebAssembly features

None. No --enable flag is needed.

Candidate fix and validation

I have a small fix that returns before accessing the output for size == 0, plus GoogleTests covering all three public functions in the regular unit-test target. The two focused regressions, all 137 WABT unit tests, a three-function ASan/UBSan harness, and the full WERROR=ON CMake build and check target pass on macOS with AppleClang.

I did not run the exhaustive 32-bit hexfloat_test input space.

Start with the public declarations in wabt/literal.h and the shared implementation used by WriteFloatHex(), WriteDoubleHex(), and WriteUint128(). Add zero-size regression coverage in the regular GoogleTest target, then run the focused tests, all 137 WABT unit tests, and the WERROR=ON CMake check target; done means none of the three functions accesses the output for a zero-sized buffer.