cactus-compute / cactus-compute/needle

[native CLI] `needle --serve` hangs in `needle_complete` spin loop on first /complete request

Open
#115 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
11k
Forks
710
Avg merge
1d 21h
Merged PRs (30d)
25

Description

## Summary
Running the native macOS CLI (`macos-arm64/needle`) in serve mode against the README's quickstart tools file (`get_weather` + `set_thermostat`) causes the process to enter a spin loop inside `needle_complete` on the very first `POST /complete` request. The connection is never accepted for subsequent requests, no response is written, and the process pegs multiple cores indefinitely.

## Environment
- Binary: `/Users/tom/Models/cactus-needle2/needle` (14,610,568 bytes, matches `Cactus-Compute/needle2` `macos-arm64/needle` byte-for-byte, dated 2026-09-07)
- OS: macOS 27.0 (Darwin 27.0.0), M-series (arm64)
- CLI flags: `--serve --port 8082 --tools /tmp/needle-tools.json`
- Tool set: the exact `get_weather` / `set_thermostat` schemas from the Quickstart section of `README.md` (see #51 for context)

## Reproduction
```sh
cd ~/Models/cactus-needle2
./needle --serve --port 8082 --tools /tmp/needle-tools.json &
# logs only the startup line — nothing further is ever written
curl -X POST http://127.0.0.1:8082/complete -d '{"input":"what is the weather in Lagos"}'
# connection times out, process never returns
```

## Observations
1. The log file (`needle-server-YYYYMMDD.log`) only contains the startup banner — no request lines, so the request never reaches a serialisation path that produces output.
2. `lsof -p ` shows the listener is bound, but the inbound TCP connection is accepted and then sits with zero bytes read/written until I kill the process. There is exactly one closed stale connection in the FD table after several minutes.
3. **RSS stays at ~14.6 MB the entire time** — no model weights are being paged in. This is purely a CPU spin, not an inference pass.
4. `sample ` shows the main thread stuck entirely inside `needle_complete`:
```
1177 Thread_93436044 DispatchQueue_1: com.apple.main-thread (serial)
1177 start (in dyld)
1177 needle_complete (in needle) + 4248
1176 ??? (in needle) load address 0x102988000 + 0xb730
1176 ??? (in needle) load address 0x102988000 + 0x61e8c
1176 ??? (in needle) load address 0x102988000 + 0x447c
781 ??? (in needle) load address 0x102988000 + 0x46944
781 ??? (in needle) load address 0x102988000 + 0x496b8
210 ??? (in needle) load address 0x102988000 + 0x5754c
```
The recursion depth in `needle_complete` and absence of any symbol resolution confirms this is the grammar-constrained decode loop. 1177/1177 samples are inside it — not a request-handling thread.
5. The bug burns ~380% CPU continuously until the process is killed (verified over a 24h run). No timeout, no log line, no way for the operator to know the request will never complete.

## Likely relationship to #51
Issue #51 was a Python-package bug (shared `needle_init` engine with stale tools) that produced a wrong-but-bounded response — the model reasoned about a `description` field that wasn't in the schema, then truncated with `token budget exhausted`. The same `get_weather`/`set_thermostat` schema mismatch that produced wrong-tool reasoning in #51 may be putting the *native* CLI's constrained decode loop into a state with no grammar-accepting continuation, so the engine retries forever instead of falling back to a refusal response like the Python path does.

## Expected
Either:
- a structured `refuse` / `truncated` JSON response matching the Python package's behaviour (preferred, matches #51's resolution), or
- an iteration-cap that emits an error after N decode steps so the request at least terminates.

## Actual
Process spins at ~380% CPU, never logs anything, never writes a response, never returns from `needle_complete`. Killed only by `SIGKILL`.

## Workaround
None observed. The CLI appears unusable with the README's example tools file. Limiting this to the Python package is not a documented constraint.

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the hang with the native macOS CLI, the README quickstart tools file, and the POST /complete request, then inspect the needle_complete path identified by sample. Compare the native behavior with the Python package behavior described in issue #51; done means the request terminates with a refusal, truncated response, or bounded error instead of spinning indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos
Domain
backend, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.