google-research / google-research/android_world
RecipeDeleteMultipleRecipesWithConstraint can generate vacuous deletion tasks with an empty target set
- Dominant language
- Python
- Stars
- 906
- Forks
- 186
- Avg merge
- 1h 21m
- Merged PRs (30d)
- 1
Description
## Summary
`RecipeDeleteMultipleRecipesWithConstraint` can generate task instances where the user is asked to delete recipes using an ingredient, but there are no matching recipes in the initialized Broccoli database. In the attached retained run, the goal is:
```text
Delete the recipes from Broccoli app that use spirulina in the directions.
```
The generated parameters contain `ingredient: "spirulina"` and `row_objects: []`. The 29 initialized `noise_row_objects` also do not contain `spirulina` in either `title` or `directions`. The agent therefore deleted nothing, and the native evaluator returned `success: 1.0`.
This is not the same as the mixed-case `Parmesan` bug where matching rows exist but are missed. This issue is about vacuous task generation: the task asks the agent to delete matching recipes, but the instance contains no matching target at all.
## Expected behavior
For a deletion task class with `n_rows = 3`, generated instances should normally contain at least one target recipe to delete, preferably the requested number of target rows. If no matching recipes can be generated for a selected ingredient, the generator should choose a different ingredient or reject/regenerate the instance.
Alternatively, if empty-target tasks are intentionally allowed, the user-facing goal and metadata should make that explicit, because otherwise the benchmark can award success for doing nothing on a task that appears to require deletion.
## Actual behavior in the attached run
`native_evaluator_input.json` records:
```json
"ingredient": "spirulina",
"row_objects": []
```
A direct check of the retained `noise_row_objects` finds zero rows whose `title` or `directions` contain `spirulina`.
The trajectory shows the agent inspected visible recipes such as `Beef Stir Fry`, `Cauliflower Fried "Rice"`, `Chicken Alfredo Pasta`, and `Chickpea Vegetable Soup`, found no `spirulina`, deleted nothing, and marked the task complete. The official evaluator output still reports:
```json
"success": 1.0
```
## Likely root cause
In `RecipeDeleteMultipleRecipesWithConstraint.generate_random_params`, the generator starts with `n_rows = cls.n_rows`, then decrements `n_rows` on `ValueError` until it reaches zero:
```python
targets = []
n_rows = cls.n_rows
while n_rows > 0:
try:
targets = sqlite_schema_utils.get_random_items(
n_rows,
_generate_random_recipe,
replacement=False,
filter_fn=lambda r: ingredient in r.directions.lower(),
)
break
except ValueError:
n_rows -= 1
return {
sqlite_validators.ROW_OBJECTS: targets,
sqlite_validators.NOISE_ROW_OBJECTS: noise,
'ingredient': ingredient,
}
```
`_COMMON_INGREDIENTS` also includes exotic ingredients that are "likely not in the existing recipes", including `spirulina`. When such an ingredient is selected and no matching recipes exist, the loop can return an empty `targets` list. The task then becomes a no-op, and the deletion validator can pass because there are no target IDs to remove.
## Suggested fix
Reject empty target sets for this task class. For example:
```python
if not targets:
raise ValueError(
f'Could not generate any target recipes for ingredient {ingredient!r}.'
)
```
or retry ingredient selection until at least one target row exists:
```python
for _ in range(max_attempts):
ingredient = random.choice(_COMMON_INGREDIENTS)
ingredient_lower = ingredient.lower()
targets = ...
if targets:
break
else:
raise ValueError('Could not generate a non-empty target set.')
```
If the intended invariant is exactly `cls.n_rows` targets, the generator should require `len(targets) == cls.n_rows` instead of silently reducing to fewer rows.
## Files in the evidence package
- `evidence/native_evaluator_input.json`: shows `ingredient: "spirulina"` and `row_objects: []`.
- `evidence/native_evaluator_output.json`: shows `success: 1.0`.
- `evidence/trajectory_steps.json`: shows the agent inspected recipes, found no spirulina, and completed without deletion.
- `source/recipe.py`: local copy of the relevant AndroidWorld task source.
- `source/sqlite_validators.py`: local copy of the deletion validator used by the task.
[RecipeDeleteMultipleRecipesWithConstraint_agent_b_spirulina_empty_target_issue.tar.gz](https://github.com/user-attachments/files/27901567/RecipeDeleteMultipleRecipesWithConstraint_agent_b_spirulina_empty_target_issue.tar.gz)
Contributor guide
Assessment
This issue has not been assessed yet.