facebookresearch / facebookresearch/matrix
Add optional acceptance/provenance gates for synthetic multi-agent outputs
- Dominant language
- Python
- Stars
- 291
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
## Motivation
Matrix is positioned as a scalable framework for multi-agent synthetic data generation, with data curation, quality filtering, and augmentation as first-class use cases.
From the current agent pipeline, quality checks appear to be implemented mostly inside task-specific agents and metrics accumulators. For example, `matrix/agents/examples/nr_curation.py` has filter/score/question agents that set `success` and `termination_reason`, and `matrix/agents/sink.py` can write only successful orchestrators via `output.success_only`.
That is useful, but it leaves a common gap for large-scale synthetic data generation: accepted rows and rejected rows do not appear to have a reusable output-boundary gate with structured provenance, negative-control checks, and auditable rejection reasons.
At Matrix scale, silently accepting plausible but unsupported synthetic rows can be more damaging than simply losing a row.
## Proposal
Add an optional reusable acceptance/provenance gate that runs before `Sink` writes a synthetic data row.
Conceptually:
```text
multi-agent trajectory
-> candidate synthetic row
-> acceptance/provenance gate
-> accepted output JSONL
-> rejected/audit JSONL with structured reasons
```
A minimal interface could look like:
```python
@dataclass
class AcceptanceGateResult:
accepted: bool
reasons: list[str]
scores: dict[str, float]
negative_control_hits: int
provenance: dict[str, Any]
```
And a config surface could be:
```yaml
output:
path: out.jsonl
rejected_path: rejected.jsonl
success_only: true
acceptance_gate:
_target_: matrix.agents.gates.CompositeAcceptanceGate
```
## Possible checks
A first implementation could stay conservative and only standardize the boundary, without changing any task-specific semantics:
1. existing `orchestrator.is_success()` result
2. schema validity of the final row
3. required fields / answer format checks
4. per-agent provenance, including which agent produced which accepted field
5. duplicate or near-duplicate signals
6. optional negative-control probes
7. explicit rejection reasons written to an audit file
## Why negative controls matter
For multi-agent synthesis, a row can look locally successful while still being unsupported in a broader sense: for example, a critic/score agent may reward format compliance, while the generated row leaks source text, duplicates a prior row, or passes an unintended control prompt.
A small local validation pattern I have found useful is:
```text
training-derived candidate
-> compare cold vs candidate behavior on held-out frontier
-> accept only if held-out behavior improves
-> reject if unsupported probes begin passing spuriously
```
Translated to Matrix, this would not require a new data-generation algorithm. It would simply give users a standard place to attach task-specific gates and preserve rejected trajectories for audit.
## Suggested minimal PR shape
1. Add `matrix/agents/gates.py` with `BaseAcceptanceGate`, `NoOpAcceptanceGate`, and maybe `CompositeAcceptanceGate`.
2. Add optional `output.acceptance_gate` and `output.rejected_path`.
3. Call the gate in `Sink.preprocess()` before writing accepted output.
4. Preserve rejected rows with `gate_result` metadata.
5. Add unit tests showing that a row with `success=True` can still be rejected by a configured gate and routed to the rejected audit file.
This would complement the existing task-specific agents rather than replace them.
Contributor guide
Assessment
This issue has not been assessed yet.