iris_execute output capture drops non-ASCII characters — tmpfile Open/Stream translation table isn't UTF-8 (present on 1.4.1)

Summary

iris_execute's temp-file output capture path drops every non-ASCII character (Japanese, Chinese, emoji) into ?. iris_query, iris_doc, and SQL persistence handle the same characters correctly on the same instance, so the issue is isolated to the output capture pipeline in build_exec_class.

This is the same file:line neighbourhood previously touched by #12, #55, and #56 — the tmpfile-based capture that iris_execute, iris_production, and iris_test all share. Those three issues fixed the shape of the pipe (multi-line quit, empty output, /tmp on Windows); this one is about its encoding.

Steps to reproduce

{
  "tool": "iris_execute",
  "args": { "code": "Write \"ASCII_ok=\", 42, !, \"nihongo=\", \"日本語テスト🍣\", !" }
}

Expected:

{"success": true, "output": "ASCII_ok=42\nnihongo=日本語テスト🍣"}

Actual (measured live on 1.4.1+260d9d4):

{"success": true, "output": "ASCII_ok=42\nnihongo=????????"}

All non-ASCII code points collapse to ?. Verified across hiragana, katakana, kanji, CJK Extension B (𠮷), and emoji including surrogate pairs.

What is unaffected (isolates the bug to the tmpfile pipe)

Tool / path	Non-ASCII round-trip	Verdict
iris_query (SELECT '日本語' AS t)	preserved	✅
iris_doc get / put a .cls with Japanese comments and string literals	preserved	✅
Persistent %String property save + iris_query read-back	preserved	✅
$System.Status.GetErrorText() of an error built with a Japanese message, then Write	mangled at the Write step, Status itself is fine	❌

So the IRIS instance is Unicode-enabled and the wire is UTF-8 clean; the drop is inside iris_execute's device redirection, not on either side of it.

Environment

• iris-agentic-dev 1.4.1+260d9d4 (debug build from master, worktree clean at time of measurement)
• IRIS: 2026.1 Build 235U Community Edition, Ubuntu ARM64 container (intersystemsdc/irishealth-community:latest, $SYSTEM.Version.IsUnicode()=1)
• Namespaces tested: USER, HSCUSTOM
• Host: macOS Darwin 25.3.0 (aarch64), Docker Desktop
• Windows / x86_64 not verified — the defect is on the ObjectScript side (device translation table), so the same behaviour is expected there

Root cause (crates/iris-agentic-dev-core/src/iris/connection.rs:707-739)

build_exec_class opens the capture tmpfile with no translation table specified and reads it back through %Stream.FileCharacter also without a translation table — both sides of the pipe silently fall back to the process default (8-bit / RAW on Unicode IRIS), so Unicode strings written by user code are transcoded to 8-bit and non-representable code points are lost:

Open tmpfile:("WNS"):5         // line 709 — no /IOT="UTF8"
Use tmpfile
// user code Writes here
...
Set stream = ##class(%Stream.FileCharacter).%New()
Set sc = stream.LinkToFile(tmpfile)   // line 737 — no TranslateTable="UTF8"
While 'stream.AtEnd { Set out = out _ stream.ReadLine() _ $Char(10) }

Decisive observation from the field: forcing the write side alone (Use $IO:(/IOT="UTF8") before the user Write) turns 日本語 into æ¥æ¬èª in the returned output — i.e. UTF-8 bytes read back through an 8-bit stream. Both sides must agree on UTF-8 or neither.

Suggested fix

Set the translation table explicitly on both ends of the tmpfile pipe:

Open tmpfile:("WNS":/IOT="UTF8"):5    // line 709
...
Set stream = ##class(%Stream.FileCharacter).%New()
Set stream.TranslateTable = "UTF8"    // insert before LinkToFile at line 737
Set sc = stream.LinkToFile(tmpfile)

Rationale for UTF8 specifically (not RAW or Native):

• IRIS ClassMethod strings are 16-bit-capable; UTF8 is the only IRIS translation table that round-trips the full Basic-Multilingual-Plane + surrogate-pair range through a byte-oriented tmpfile.
• Atelier v8 response bodies are already UTF-8, so no additional Rust-side conversion is required — the value that ends up in the SQL response is a well-formed IRIS string with the original code points intact.
• The change is invisible to ASCII-only Write output (no regression risk for the existing English test surface).

Alternatively, switch to %Stream.FileBinary on both sides and defer the UTF-8 decode to Rust; less risk of unrelated IRIS-side quirks, but requires touching the Atelier response handling too. The /IOT="UTF8" approach is smaller.

Verified locally: 2-line diff on crates/iris-agentic-dev-core/src/iris/connection.rs (one Open param, one stream.TranslateTable), rebuilt debug binary. All Unicode planes round-trip end-to-end:

Input	Before	After
Write "ひらがな",!	????	ひらがな
Write "漢字",!	??	漢字
Write "🍣🍺",!	????	🍣🍺
Write "𠮷野家",! (CJK-B surrogate)	?????	𠮷野家

ASCII / multi-line / empty-output regression cases were re-run and are clean (Write "HELLO",! → HELLO; Write $ZV,! → full version string per the tool docstring; Write "line1",! Write "line2",! → line1\nline2; Set x = 1 with no Write → "").

Existing build_exec_class_* unit tests continue to pass unchanged.

Regression test suggestion

Add a Japanese/emoji Write assertion to the iris_execute integration tests (--features testing, #[ignore] for the live-IRIS group). Suggested minimal case:

// crates/iris-agentic-dev-core/tests/integration/iris_execute.rs
#[tokio::test]
#[ignore] // live IRIS required
async fn iris_execute_preserves_non_ascii_write_output() {
    let out = call_iris_execute(r#"Write "日本語テスト🍣""#).await;
    assert!(out.contains("日本語テスト🍣"), "mojibake: {out}");
}

Also add a build_exec_class unit test that asserts the emitted ObjectScript contains /IOT="UTF8" on the Open line and TranslateTable="UTF8" before LinkToFile.

Related

• #12 — multi-line output → invalid Quit (same build_exec_class, output content shape)
• #55 — silent empty output (same build_exec_class, capture-side wiring)
• #56 — hardcoded /tmp (same build_exec_class, platform portability)

Workaround (interim)

iris_query "SELECT '日本語' AS t" and iris_doc get/put round-trip Japanese correctly, so most read-back needs can be routed through them. But Write-based progress/error messages have no in-band workaround.

---

Happy to open a PR with the 2-line ObjectScript diff, the build_exec_class unit test asserting the /IOT="UTF8" and TranslateTable="UTF8" lines, and the #[ignore]d live-IRIS integration test.

Start in crates/iris-agentic-dev-core/src/iris/connection.rs around build_exec_class, especially lines 707-739, then inspect its existing unit tests. Run the build_exec_class tests and, with a live IRIS instance and the testing feature enabled, run crates/iris-agentic-dev-core/tests/integration/iris_execute.rs. Done means non-ASCII Write output round-trips while the existing ASCII, multiline, empty-output, and version cases remain passing.