google / google/langfun

Security: default lf.query Python protocol executes model output via exec() (CWE-94/95)

Open
#725 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
915
Forks
81
Avg merge
21d 21h
Merged PRs (30d)
7

Description

## Summary

`lf.query(prompt, schema, lm=...)` parses the model's response into a Python object. On the default protocol (`protocol='python'`), the parse step executes the model's response as Python via `pyglove.coding.evaluate`, with `sandbox=False`. The only guard is an AST node-type filter (`CodePermission.ASSIGN|CALL`) that restricts node *types* but not which names or builtins may be called. As a result, attacker-influenced model output can reach `exec()` on default settings — i.e. **anything the model can be made to say, langfun will run as Python in the host process.**

This was reported to Google's OSS VRP (issue tracker [518137699](https://issuetracker.google.com/issues/518137699)) and classified **Won't Fix (Intended Behavior)**; the triager acknowledged it as a product vulnerability but noted langfun's tier makes it ineligible for a reward and suggested raising it here directly. Filing this as a public, constructive record with a proposed fix. Quoting triager's comments below for reference:
>Hey there! Thanks for your report on the langfun project regarding the eval injection leading to RCE via lf.query's default 'python' protocol. You've clearly identified a product vulnerability here.
>While langfun is indeed a Google open-source project and falls under the OSS VRP, our program rules state that product vulnerabilities in OT2 (Standard) and OT3 (Low-Priority) tiered repositories are not eligible for monetary reward. The langfun repository currently falls into one of these tiers (as 'TIER_UNSPECIFIED' implies).
>This means we won't be able to offer a reward for this specific finding, but we definitely appreciate you bringing it to our attention. You're more than welcome to open an issue or submit a pull request directly to the langfun GitHub repository to help get this fixed.

## Impact: why this is a vulnerability, not just code execution by design

This is remote code execution. In any deployment where the model's input can be influenced by an attacker — a RAG app over user-supplied documents, an agent processing tool output or web pages, anything summarizing untrusted text — a successful prompt injection turns into **arbitrary Python execution inside the application's process.** Concretely, an attacker who lands the payload can:

- read process secrets (API keys, DB credentials, tokens in env/memory),
- read and write the filesystem with the app's privileges,
- make outbound network requests (exfiltration, SSRF, pivoting to internal services),
- and otherwise run anything the host process can run.

The execution is silent: the malicious response *also* returns a valid schema object, so the calling application receives exactly what it expected and the side effect is invisible. There is no error, no log, nothing for the developer to notice.

The reason this is a vulnerability and not merely "the Python protocol runs code" is that **it is the default, it is undocumented at the call site, and the one control that looks like a sandbox does not work.** A developer who follows the README — `lf.query("...", MySchema, lm=...)` — has not opted into executing model output; they asked the library to parse a structured object. Nothing in the call signature or docstring indicates `exec()` is involved, and the AST `permission` filter that *appears* to be a security boundary fails open (see below). "The model is trusted" is not a safe assumption — prompt injection from untrusted content in the prompt is the baseline threat for LLM applications, and treating model output as trusted Python source is the vulnerability.

## Affected

- PyPI `langfun` **0.1.1** (latest release; full history is 0.0.1 / 0.1.0 / 0.1.1) — confirmed
- `main` @ `e80093b` — confirmed
- No hardening of this path is present on `main` (the only structured-parser commit since 0.1.1 is refactor `b1d6d95`, which moved the file but preserved `sandbox=False` and the `ASSIGN|CALL` default).

## Resolved defaults on a no-args `lf.query` call

```
protocol='python', sandbox=False, permission=ASSIGN|CALL (value 9)
```

- `querying.py:466-467` — protocol default resolves to `'python'`
- `schema/python.py:167` — `sandbox=False`
- `schema/python.py:142-144` — permission default `ASSIGN|CALL`
- `schema/python.py:163` comment, verbatim: `# We are creating objects here, so we execute the code without a sandbox.`

## Call chain (public API → `exec()`)

```
lf.query -> Mapping.parse_result (mapping.py:490)
-> PythonPromptingProtocol.parse_value (schema/python.py:110-135)
-> structure_from_python(sandbox=False) (schema/python.py:138-171)
-> correction.run_with_correction (correction.py:42)
-> execution.run (execution.py:69)
-> pg.coding.evaluate -> exec(compile(...)) (pyglove execution.py:128)
```

`lf.parse` and `lf.call` reach the same sink on the same default.

## Minimal reproduction

Using a mock LM so the "model output" is supplied directly — no network, no API keys. The entire model response is a single expression that looks like an ordinary schema literal:

```python
import os, pathlib
import pyglove as pg
import langfun as lf

CANARY = pathlib.Path(f"/tmp/lf_canary_{os.getpid()}")
if CANARY.exists(): CANARY.unlink()

class Person(pg.Object):
name: str
age: int

# Single expression; no __builtins__ reference, no __import__, no lambda.
body = f"Person(name=open('{CANARY}','w').write('hit') and 'A', age=1)"
r = lf.query("prompt", Person, lm=lf.llms.StaticResponse(f"```python\n{body}\n```"))

print("result:", r) # valid Person(name='A', age=1)
print("canary written:", CANARY.exists()) # True -> code executed as a side effect
if CANARY.exists(): CANARY.unlink()
```

`lf.query` returns a valid `Person` **and** writes the canary file as a side effect. `write()` returns an int (truthy), so ` and 'A'` short-circuits to `'A'` and the schema validates normally.

> **Reproduction note:** verify via the filesystem canary, not via `print`. `pg.coding.evaluate` wraps the `exec` in `contextlib.redirect_stdout` (`pyglove execution.py:104`), so a print-based check appears to do nothing even when the code executed.

## Why the AST filter does not prevent this

The `permission=ASSIGN|CALL` filter allows `ast.Call` and all expression nodes, but it does not constrain *which* callables are invoked. CPython auto-populates `__builtins__` into the `exec` globals, so `open` and `__import__` are reachable by bare name (both confirmed). Removing `__builtins__` is not a complete fix — it auto-repopulates, and the schema's own classes plus `pg` itself remain reachable with arbitrary `Call` nodes permitted. The filter restricts node *types*, not node *contents*; as a code-execution boundary it does not hold.

## Threat model

An application built on langfun (RAG, agent, document processing) feeds content into the prompt that an attacker can influence — an ingested document, a user message, a scraped page, a tool output, a retrieved email. Prompt injection steers the model to emit the expression above, and on default settings langfun executes it in the application's process — yielding arbitrary code execution with the application's privileges (secret theft, filesystem access, outbound network). No host access, no knowledge of the schema, and no non-default configuration is required.

## Proposed remediation (in priority order)

1. **Replace `exec`-based parsing with a literal-only / schema-aware evaluator** — an `ast.literal_eval`-style walker that permits construction only of the schema's declared classes (plus stdlib containers like `list`, `dict`, `tuple`, `set`). Legitimate structured-output responses are always of the form `Schema(field=...)` with literal or nested-schema values; they never require `Call` on free names like `open` or `__import__`. This removes the sink entirely.
2. **Interim hardening:** default `sandbox=True` in `structure_from_python` and fail loudly if a sandbox cannot be created.
3. **Document the behavior:** note in the `lf.query` / `lf.parse` / `lf.call` docstrings and a README "Security model" section that the Python protocol executes model output, and consider a runtime warning until (1) lands.

I'm happy to open a PR, if that's a welcome direction.

> **Note on the JSON protocol:** the non-default `protocol='json'` path is not a safe drop-in alternative — it routes to `pg.from_json_str(auto_import=True)` in pyglove, which imports attacker-named modules and, for `_type: "function"`, runs `marshal.loads(...)`. That is a separate concern that belongs upstream in `google/pyglove`. Flagging only so "switch the default to JSON" isn't read as a fix on its own.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.