emscripten-core / emscripten-core/emscripten

ptrToIdx() and idxToPtr() macros to unify 2GB, 4GB and Wasm64 programming

Open
#25,600 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
27.6k
Forks
3.6k
Avg merge
1d 1h
Merged PRs (30d)
105

Description

In the old PR https://github.com/emscripten-core/emscripten/pull/19289 I proposed three helper functions `{{{ ptrToIdx() }}}`, `{{{ convertPtrToIdx() }}}` and `{{{ idxToPtr() }}}` to provide a unified programming model for converting between Wasm pointers and pre-shifted typed array heap indices.

There was some pushback in the PR, so the feature did not land.

I ended up landing the feature into Unity's fork of Emscripten, relegating to the idea that I would make those a Unity .js library `{{{ macro }}}`s.

But later on I realize that it is not actually possible for Emscripten users to generate "system level" JS library wide `{{{ }}}` macros, the parsing order is a PITA to deal with.

So I would like to gauge the temperature of whether we might revisit landing `{{{ ptrToIdx() }}}`, `{{{ convertPtrToIdx() }}}` and `{{{ idxToPtr() }}}` as end-user programmable macros into Emscripten?

Given that Wasm produces pointers as 64-bit BigInts, or int32s that need to be `>>> 0`d in 4GB mode, it would be ideal to have one programming model to cover all of these cases.

Currently Emscripten has ad hoc `#if CAN_ADDRESS_2GB` handling in some call sites:

https://github.com/emscripten-core/emscripten/blob/6ca370067cacb6a6102ed7245c611660ab38a514/src/lib/libstrings.js#L118-L120

and ad hoc `#if MEMORY64` handling in others:

https://github.com/emscripten-core/emscripten/blob/6ca370067cacb6a6102ed7245c611660ab38a514/src/lib/libccall.js#L54-L56

along with `to64()` and `from64()` macros that perform pointer<->heap index conversion in Wasm64 mode only.

The automatic `foo__sig: '....'` model helps automate this challenge in some cases, but it has [undesirable interactions with JSPI](https://github.com/juj/wasm_webgpu/blob/54c51d715d68f22a62e1f84eb072737e8a8270c0/lib/lib_webgpu.js#L672), and it results in pessimization of generated code size.

In the [juj/wasm_webgpu](https://github.com/juj/wasm_webgpu) library, I ended up needing the following constructs for most convenient handling of 2GB+4GB+Wasm64: https://github.com/juj/wasm_webgpu/blob/54c51d715d68f22a62e1f84eb072737e8a8270c0/lib/lib_webgpu.js#L26-L88

(where the functions are a bit awkwardly named to avoid possible name conflict that I was anticipating with #19289)

Would there be interest in revisiting the idea of adding `{{{ ptrToIdx() }}}`, `{{{ convertPtrToIdx() }}}` and `{{{ idxToPtr() }}}` macros? I would like to be able to provide a single documented API to Unity users so that they can manage 2GB, 4GB and Wasm64 programming models in a unified fashion.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.