Snapchat / Snapchat/Valdi

getModuleFileEntryAsBytes / .bin byte-exact bundled data does not work on web (silent wrong data, not an error)

Open
#128 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
16.4k
Forks
537
Avg merge
22d 5h
Merged PRs (30d)
1

Description

Summary

The byte-exact bundled-data mechanism added for getModuleFileEntryAsBytes/.bin srcs files (docs: docs/docs/advanced-images.md#byte-exact-raw-bitmap-data, example: apps/helloworld/src/valdi/hello_world/src/ByteExactImageExample.tsx) does not work on the web target, and fails silently rather than erroring.

The docs state:

getModuleFileEntryAsBytes returns the file's exact bytes, on every platform.

This is not true for web.

Root cause

Two separate gaps, at two different layers:

  1. Compiler: .bin srcs files are never emitted into the web build output at all. I confirmed this empirically — after building a consuming app's web target, its .ayab_web_cache output directory (the Valdi-compiled JS/asset tree consumed by webpack) contains zero .bin files anywhere, even though the same module builds and bundles correctly for iOS/Android/macOS. Looking at bzl/valdi/valdi_compiled.bzl, _get_srcs_js_paths only maps .ts/.tsx/.js sources to .js web outputs — any other extension is silently skipped and never declared as a web output. The .valdimodule archive packing that makes .bin retrievable at runtime (_will_generate_valdimodule, valdi/src/valdi/runtime/Resources/Bundle.cpp) is only wired for iOS/Android/native/standalone targets; there is no web equivalent. (There is a working precedent for "verbatim-copy a raw file into a per-platform web output path" — the strings-JSON pipeline, e.g. _get_web_string_resource_paths in the same file — but that copy is performed by the Swift local_valdi_compiler binary itself as part of its declared ctx.actions.run outputs, not by a Bazel/shell step, so extending it to .bin requires compiler-side (Swift) changes, not just Bazel/TS.)

  2. Runtime: getModuleEntry in src/valdi_modules/src/valdi/web_renderer/src/ValdiWebRuntime.ts (~line 293) is a hardcoded stub:

    // Stubbed — was using jsonContext (removed). Localized strings now use
    // _strings_preload.js generated by collapse_web_paths instead.
    getModuleEntry(module: string, pathStr: string, asString: boolean) {
      return '{}';
    }
    

    It always returns the literal string '{}' regardless of arguments, and never throws. getModuleFileEntryAsBytes casts this to Uint8Array, so callers get back the 2 bytes {/} with no error, no warning — code that then tries to decode those bytes as image data (e.g. decodeBitmap) doesn't throw either; it just silently produces a bogus/degenerate result (in our repro, a 1x1 bitmap instead of the real 60x10 source image). This is the more actionable half of the bug: even without the compiler-side fix, getModuleEntry should fail loudly on web instead of returning stub data that downstream code can't distinguish from success.

Repro

  1. Add a module with a .bin src per the advanced-images.md byte-exact pattern (srcs = glob([..., "src/**/*.bin"]), inline_assets = True).
  2. Call getModuleFileEntryAsBytes(module, path) for that file from a component that also renders on web (e.g. via webpack serve / npm start in a consuming app's web/ dir).
  3. Native builds (iOS/Android/macOS-standalone bazel test) return the correct bytes. The web build returns Uint8Array bytes for the string '{}' instead, and no error is surfaced anywhere (console, exception, or otherwise).

Suggested fix

  • Short term (low risk, high value): make getModuleEntry's web stub throw instead of returning '{}', so this fails loudly instead of masquerading as success. This alone would have made the underlying bug in our app immediately obvious instead of costing significant debugging time to trace through decoded-bitmap-looks-wrong -> traced back through three layers to this stub.
  • Longer term: either (a) extend the compiler to emit .bin srcs verbatim into a web output path (mirroring the strings-JSON precedent) plus a webpack-visible registry (mirroring the existing _image_registry.js / _strings_preload.js generation in bzl/valdi/valdi_collapse_web_paths.bzl), giving .bin real web support; or (b) if web support isn't planned, update advanced-images.md to explicitly scope the "on every platform" claim to native platforms only, so consumers don't reach for this API expecting web parity.

Environment

  • Found via a downstream consumer app pinned to Valdi SHA 919f23c8ebdd3582b58821a9ff252b93ead5eb75 (the commit that introduced the byte-exact .bin feature / ByteExactImageExample.tsx).
  • Repro path: web dev server via webpack serve (webpack 5), bazel test (native JS engine / valdi_standalone) does not reproduce this — it only shows up in an actual browser/webpack build.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/valdi_modules/src/valdi/web_renderer/src/ValdiWebRuntime.ts at getModuleEntry and reproduce with the consuming app's web target via webpack serve. Then inspect bzl/valdi/valdi_compiled.bzl and the Swift local_valdi_compiler path for .bin handling. Done means the chosen web-support scope is explicit and the ByteExactImageExample.tsx flow no longer silently receives bogus data.

Written by the indexing model from the issue text.

Assessment

Tech stack
swift, typescript, webpack
Domain
build-system, tooling, web-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.