aws-samples / aws-samples/aws-batch-comp-infrastructure-sample
`test-local` always reports that no result is produced by distributed run
- Dominant language
- Python
- Stars
- 12
- Forks
- 14
- Avg merge
- 16h 1m
- Merged PRs (30d)
- 3
Description
On the `mainline-2026` branch, I believe that `test-local` is wrong for distributed solvers, implemented in `DistributedTestRunner`.
The documentation states that the function `get_solver_result` in `solver_cmd.py` is the same for both parallel and distributed version, but the member function `_collect_results` of `DistributedTestRunner` seems to parse JSON within the output log to determine the result, while `TestLocalCommand` does not.
It also seems to me that `DistributedTestRunner` never calls `get_solver_result`.
When `_collect_results` processes each output message, it outputs `NO_RESULT` as a fallback when no result is recognized, which happens for all test cases for me.
By contrary, `test-local` worked fine for me when I configured the solver as non-distributed (in which case `DistributedTestRunner` is not called).
I also experienced that when running `test-local` with `--results-dir`, the `stdout.log` contains log messages of the leader node, such as:
```
2026-08-08 21:40:49 - Entrypoint - INFO - Local IP address: 172.18.0.9
2026-08-08 21:40:49 - Entrypoint - INFO - Environment variables are: {'SOLVER_NAME': 'smts', 'PROJECT_NAME': 'smts26', 'SOLVER_NODE_TYPE': 'distributed-leader', 'NUM_WORKERS': 7, 'LOCAL_TEST_FILES': '/opt/amazon/test_formulas/**/*.smt2', 'LOCAL_TIMEOUT': 60}
2026-08-08 21:40:49 - Entrypoint - INFO - I am a leader. About to call `leader.run()`
2026-08-08 21:40:49 - Leader Entrypoint - INFO - Started leader entrypoint
2026-08-08 21:40:49 - Leader Entrypoint - INFO - Attempting to claim 7 workers...
2026-08-08 21:40:49 - Leader Entrypoint - INFO - No crashed/dead nodes to clean up
2026-08-08 21:40:49 - Leader Entrypoint - INFO - Found 0 workers already claimed by me (out of 7 needed workers)
2026-08-08 21:40:49 - Leader Entrypoint - INFO - We don't have enough workers. Claiming more...
2026-08-08 21:40:49 - Leader Entrypoint - INFO - Claimed the following 7 workers: [IpItem(2fe50008-b1dd-4295-b158-4d5b39a585d1), IpItem(d6d4c02e-aae7-4a46-aeb5-d02a495db79e), IpItem(04eef2e2-8be6-44e9-b75a-7b7eb262472a), IpItem(225c6491-38cd-4ff2-a060-ec0bcfb33aa3), IpItem(2a763df9-2d08-45ed-8a31-41647a7704bc), IpItem(7f422bf9-a91d-46d2-873d-5612c40bc8da), IpItem(a545f905-c294-40bf-8671-4645513e8ab6)].
2026-08-08 21:40:49 - Leader Entrypoint - INFO - Checking that all workers are healthy
2026-08-08 21:41:09 - Leader Entrypoint - INFO - No message on the queue, trying again
2026-08-08 21:41:29 - Leader Entrypoint - INFO - No message on the queue, trying again
2026-08-08 21:41:49 - Leader Entrypoint - INFO - No message on the queue, trying again
2026-08-08 21:41:49 - Leader Entrypoint - INFO - Exiting the leader entrypoint
2026-08-08 21:41:49 - Leader Entrypoint - INFO - Here are the output queue messages:
```
while it contains just `sat` or `unsat` when `test-local` is run in the parallel (non-distributed) mode.
Is `DistributedTestRunner` malformed or outdated, or am I doing something wrong?
Contributor guide
Research direction
Start with get_solver_result in solver_cmd.py and DistributedTestRunner._collect_results, then reproduce test-local with a distributed solver and compare its output with the parallel mode. Done means distributed runs recognize the solver result instead of reporting NO_RESULT, while --results-dir output reflects the expected test output rather than only leader-node messages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100