Difficulties with internally using refined types

We currently use or plan to use more refined types in the IR than we will eventually emit in the binary in a couple situations:

• Function references always use concrete signature types rather than funcref even when GC is not enabled.
• Strings use stringref rather than externref even when stringref is not enabled.
• We plan to use exact heap types where possible even when custom descriptors are not enabled.

In each of these cases, we wish to use the more refined types internally because the extra type information can in principle help us optimize better. The binary writer generalizes the internal refined types to their most precise allowed supertype when writing the binary to ensure that our output only uses the allowed features.

However, type generalization in the binary writer is the source of many latent bugs that have not been found by the fuzzer or encountered by users.

• Casts distinguishing values with refined types from their unrefined supertypes will no longer be able to distinguish these values after their target types have been generalized. Optimizations in OptimizeInstructions, RemoveUnusedBrs, Precompute, and GUFA based on the assumption that these casts fail are incorrect.
• MinimizeRecGroups and TypeUpdater more generally depend on being able to distinguish rec group structures to ensure that separate types remain separate after optimizations. If the rec groups only differ because one uses a more refined type that will be generalized during binary writing, the types will no longer be separate in the final binary.

These bugs have not been found because they either depend on string lowering, which is not yet fuzzed, or depend on GC being enabled so that casts exist to be optimized, etc. Using exact types with GC modules and optimizations without enabling custom descriptors will surface these bugs.

The plan to fix these bugs is to update the utilities used by these optimizations to evaluate cast results and compare rec group structures to take the enabled features into account and apply the same generalization logic that will eventually be applied in the binary writer.

An alternative plan would be to not use refined types internally when that could lead to bugs. The relative merit of this approach will depend on how much extra optimizing power using the refined types ends up unlocking.

Start with the binary writer’s type-generalization logic and the utilities used by OptimizeInstructions, RemoveUnusedBrs, Precompute, GUFA, MinimizeRecGroups, and TypeUpdater. Compare their cast-result and rec-group comparisons against enabled features; done means these optimizations remain correct when refined types are generalized, including with GC modules and custom descriptors disabled.