cactus-compute / cactus-compute/needle
needle.environments: five of six frozen acceptance suites fail on the shipped engine
- Dominant language
- Python
- Stars
- 11k
- Forks
- 710
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 25
Description
Hello, and thanks for shipping the environments modules — the "five tools or fewer, closed sets as enums" framing is a genuinely useful pattern to copy from.
I ran the frozen acceptance suites against the shipped engine and five of the six do not pass. I may well be holding this wrong, so please tell me if these suites are aspirational targets rather than a gate that is expected to be green today — the docs read as the latter to me, which is why I am raising it.
## What I see
`python -m needle.environments.` exits 1 for every environment except `data_capture`:
| Environment | default gate (`min_confidence=0.0`) | `min_confidence=0.4` |
| --- | --- | --- |
| `data_capture` | 30/32, 0 critical — pass | 29/32, 0 critical — pass |
| `media_player` | 27/32, 3 critical | 30/32, 0 critical — pass |
| `kitchen_appliance` | 28/32, 1 critical | 30/32, **1 critical** |
| `smart_home` | 25/32, 3 critical | 27/32, 0 critical (below 90%) |
| `wearable` | 27/32, 5 critical | 31/32, **1 critical** |
| `productivity` | 28/32, 3 critical | 24/32, **1 critical** |
So the README snippet returns `False`:
```python
from needle.environments import smart_home
smart_home.run_tests() # -> False
```
and `needle.environments.run_tests(0.4)` is also `False`.
The results are deterministic — repeated runs give identical scores — and identical on engine 2.0.3 and 2.0.4, so this does not appear to be a recent engine regression. It looks like the suites have simply never been green on the shipped base engine.
## The stubborn cases
At the 0.4 production gate the remaining critical failures are all the model acting where the suite expects a refusal:
```
kitchen_appliance [missing] "start a timer for 25 minutes"
want []
got [{"name": "set_cooking_timer", "arguments": {"label": "timer", "minutes": 25}}]
productivity [invalid] "set a timer for negative 5 minutes"
want []
got [{"name": "set_timer", "arguments": {"time_human": "5 minutes"}}]
wearable [invalid] "reply to Priya saying happy birthday"
want []
got [{"name": "reply_to_notification", "arguments": {"notification_match": "Maya", "text": "happy birthday"}}]
```
The `productivity` case is the one I found most interesting: the negative sign is dropped rather than the request being refused, so an out-of-bounds value becomes a plausible in-bounds one.
`smart_home` also shows a duplicate-call pattern in its `parallel` case, where the correct call arrives alongside a partial one:
```
smart_home [parallel] "start the vacuum in the kitchen and open the living room blinds"
want [start_robot_vacuum{action:start, room:kitchen}, control_blinds{room:living_room, action:open}]
got [start_robot_vacuum{action:start}, start_robot_vacuum{action:start, room:kitchen}, control_blinds{room:living_room, action:open}]
```
That over-firing behaviour looks adjacent to #114, which reports a high-confidence tool call on nonsense input — possibly the same underlying tendency seen from a different angle.
## Why this may have gone unnoticed
`tests/test_environments.py` is engine-gated, and `test_smart_home_smoke` only asserts on a single `complete()` call rather than running a suite. As far as I can tell nothing in the release workflow runs the suites at all, so a drift here would not surface in CI.
## What I am unsure about
`doc/environments.md` describes `run_tests()` as returning `True` at ">=90% pass with zero critical failures" and `python -m needle.environments.smart_home` as "exit 0 on pass", against "the shipped engine", with no mention of fine-tuning being a prerequisite. If the intent is that these become green only after `needle finetune` on the environment's tools, a line saying so in the docs would have saved me the confusion, and I would be glad to send that doc PR instead.
## Environment
- `cactus-needle` from source at `53df049` (same content as v2.0.13), Python 3.11, Linux x86_64
- Engine 2.0.4 (`manylinux2014_x86_64`), base weights, no adapter
- Cross-checked on engine 2.0.3 with identical scores
Happy to share the full per-case output for any environment if that would be useful.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the six commands shown in the report and reviewing tests/test_environments.py, doc/environments.md, and the environment suite entry points. Compare the shipped-engine results with the documented pass criteria and inspect the failing cases, including the references to #114. Done means the suites either meet the stated gate with coverage in the release workflow, or the documentation clearly states that fine-tuning is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100