FEAT expose mask parameter on OpenAIImageTarget for inpainting support
@romanlutz is already working on this.
Since Jun 15, 2026.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
## Summary
OpenAIImageTarget._send_edit_request_async currently calls client.images.edit(**image_edit_args) with only `image`, `prompt`, `size`, and a few formatting flags. The OpenAI SDK's images.edit endpoint also accepts an optional `mask` parameter — a PNG whose fully-transparent regions indicate the only pixels the model is permitted to modify, while every other pixel must be returned unchanged. This is inpainting, and it's the standard primitive for "keep this scene but regenerate just the masked subregion."
PyRIT does not expose `mask` today, so red-teamers using OpenAIImageTarget for image-editing attacks can only do whole-image edits. They cannot pin which subregion of the input image is allowed to change, which loses an important degree of control for evaluations that depend on precisely localized edits.
## Proposed change
1. Add a new `MessagePiece` data type or convention for image masks (options: a dedicated `image_mask` data type, or reuse `image_path` with a metadata flag — e.g. prompt_metadata={"is_mask": True}).
2. In OpenAIImageTarget._send_edit_request_async, separate the input pieces into base image(s) and mask. When a mask is present, add `"mask": (name, bytes, mime_type)` to `image_edit_args`.
3. Validate that mask is only attached when exactly one base image is provided (the OpenAI endpoint only supports masking a single base image — multi-image edits with a mask would need to be rejected with a clear error).
4. Update the multimodal/router components (e.g. ModalityFeedbackRouter.build_objective_input_message) so attacks that want to forward a mask alongside the adversarial-generated text + base image can do so without rebuilding the message from scratch.
5. Tests:
- Mask piece is correctly serialized and passed through to the SDK call.
- Multi-image + mask raises a validation error.
- Backward compatibility: existing single-image and multi-image edit flows (no mask) continue to work unchanged.
## API reference
OpenAI image edit endpoint: https://platform.openai.com/docs/api-reference/images/createEdit — see the `mask` parameter.
## Scope / non-goals
- No automatic mask generation from text (e.g. "mask the face") — masks are provided by the caller.
## Acceptance criteria
- A caller can construct a `Message` with `[text, image_path, mask]` and OpenAIImageTarget.send_prompt_async routes the mask correctly to `client.images.edit(..., mask=...)`.
- The unmasked portions of the input image are pixel-for-pixel preserved (verified in an integration test against a fixture image).
- Existing tests continue to pass.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.