galaxyproject / galaxyproject/planemo

planemo test crashes when workflow test specifies non-existent outputs

Open
#1,625 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
110
Forks
102
Avg merge
4d 21h
Merged PRs (30d)
13

Description

I encountered this due to a typo in an expected output, and spent far too long trying to understand what happened.

## Description
When running planemo test on a Galaxy workflow, if the test YAML file specifies expected outputs that don't exist in the workflow definition, planemo crashes with a bioblend.ConnectionError instead of producing a meaningful error.

The workflow itself completes successfully (all jobs finish), but planemo crashes during the output collection phase because Galaxy has already shut down by the time it tries to fetch metadata for the non-existent outputs.

## Expected Behavior
Planemo should produce a helpful error, maybe in the expected HTML and JSON test report
The test should be marked as failed (not crash) with a clear error message like "Expected output [output_name] not found in results."

## Actual Behavior
Planemo crashes with a bioblend.ConnectionError
No HTML or JSON test report is generated
The error message is unhelpful for diagnosing the root cause (missing output definition)

## Steps to Reproduce
Create a workflow test YAML that references outputs not defined in the workflow
Run planemo test on the workflow
Observe the crash after the workflow completes successfully

## Traceback
```
galaxy.jobs DEBUG 2026-03-04 18:17:08,772 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0] finish():
Moved
/tmp/tmp_n3szmv_/job_working_directory/000/12/outputs/dataset_3c222a94-504d-4b79-a786-3766f10ae099.dat
to /tmp/tmp_n3szmv_/files/3/c/2/dataset_3c222a94-504d-4b79-a786-3766f10ae099.dat
galaxy.jobs DEBUG 2026-03-04 18:17:08,772 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0] finish():
Moved
/tmp/tmp_n3szmv_/job_working_directory/000/12/outputs/dataset_a108e77d-c4e8-4b07-b1f3-a6213498bd79.dat
to /tmp/tmp_n3szmv_/files/a/1/0/dataset_a108e77d-c4e8-4b07-b1f3-a6213498bd79.dat
galaxy.model.metadata DEBUG 2026-03-04 18:17:08,776 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0]
loading metadata from file for: HistoryDatasetAssociation 17
galaxy.model.metadata DEBUG 2026-03-04 18:17:08,778 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0]
loading metadata from file for: HistoryDatasetAssociation 18
galaxy.jobs INFO 2026-03-04 18:17:08,790 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0] Collecting
metrics for Job 12 in /tmp/tmp_n3szmv_/job_working_directory/000/12
galaxy.jobs DEBUG 2026-03-04 18:17:08,796 [pN:main.1,p:235019,tN:LocalRunner.work_thread-0]
job_wrapper.finish for job 12 executed (27.033 ms)
╭─────────────────────────────────── Invocation <6edbfb9fe94476e2> ────────────────────────────────────╮
│ Steps ◆ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ◆ 100% ◆ 8/8 scheduled │
│ Jobs ◆ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ◆ 100% ◆ 9/9 terminal │
│ Job States ◆ 🟢 9 ◆ │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────╯
galaxy.jobs.runners.util.process_groups DEBUG 2026-03-04 18:17:12,677 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] check_pg(): No process found in process group 235556
galaxy.jobs.runners.local DEBUG 2026-03-04 18:17:12,677 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] execution finished: /tmp/tmp_n3szmv_/job_working_directory/000/14/galaxy_14.sh
galaxy.jobs DEBUG 2026-03-04 18:17:12,689 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] finish(): Moved /tmp/tmp_n3szmv_/job_working_directory/000/14/outputs/dataset_60bd1354-66ea-41e8-b578-243dd35e2f76.dat to /tmp/tmp_n3szmv_/files/6/0/b/dataset_60bd1354-66ea-41e8-b578-243dd35e2f76.dat
galaxy.jobs DEBUG 2026-03-04 18:17:12,689 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] finish(): Moved /tmp/tmp_n3szmv_/job_working_directory/000/14/outputs/dataset_dfc07a6f-def0-4ab8-b0cb-42552ece2386.dat to /tmp/tmp_n3szmv_/files/d/f/c/dataset_dfc07a6f-def0-4ab8-b0cb-42552ece2386.dat
galaxy.model.metadata DEBUG 2026-03-04 18:17:12,694 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 21
galaxy.model.metadata DEBUG 2026-03-04 18:17:12,696 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] loading metadata from file for: HistoryDatasetAssociation 22
galaxy.jobs INFO 2026-03-04 18:17:12,709 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] Collecting metrics for Job 14 in /tmp/tmp_n3szmv_/job_working_directory/000/14
galaxy.jobs DEBUG 2026-03-04 18:17:12,716 [pN:main.1,p:235019,tN:LocalRunner.work_thread-3] job_wrapper.finish for job 14 executed (30.239 ms)
collecting outputs to directory /tmp/tmpxuy4y_tz
collecting outputs to directory /tmp/tmpaxgloio7
collecting outputs to directory /tmp/tmpzouob_or
collecting outputs to directory /tmp/tmpj0w558qa
collecting outputs to directory /tmp/tmpda6whi0l
collecting outputs to directory /tmp/tmp4id1up7i
None
Shut down
supervisord has terminated
collecting outputs to directory /tmp/tmptbuj17tg
collecting outputs to directory /tmp/tmpv7_2s_gs
collecting outputs to directory /tmp/tmpnyzrnqxc
collecting outputs to directory /tmp/tmp88com8i7
collecting outputs to directory /tmp/tmpep5r3oot
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 9 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 8 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 7 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 6 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 5 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 4 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 3 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 2 attempts left
bioblend WARNING: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 1 attempts left
bioblend ERROR: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 0 attempts left
Traceback (most recent call last):
File "/home/danielle/Documents/veg/tools-iuc/planemo/bin/planemo", line 8, in
sys.exit(planemo())
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 1485, in __call__
return self.main(*args, **kwargs)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 1406, in main
rv = self.invoke(ctx)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 1873, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 1269, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/decorators.py", line 93, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/click/core.py", line 824, in invoke
return callback(*args, **kwargs)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/cli.py", line 146, in handle_blended_options
return f(*args, **kwds)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/commands/cmd_test.py", line 76, in cli
return_value = test_runnables(ctx, runnables, original_paths=uris, **kwds)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/engine/test.py", line 13, in test_runnables
test_data = engine.test(runnables, test_timeout=kwds.get("test_timeout"))
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/engine/interface.py", line 83, in test
test_case_data = test_case.structured_test_data(run_response)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/runnable.py", line 377, in structured_test_data
return run_response.structured_data(self)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/runnable.py", line 613, in structured_data
output_value = self.get_output(output_id)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/galaxy/activity.py", line 692, in get_output
self._outputs_dict[output_id] = self.collect_outputs(ignore_missing_output=True, output_id=output_id)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/galaxy/activity.py", line 639, in collect_outputs
cwl_output = output_to_cwl_json(
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/galaxy/tool_util/cwl/util.py", line 519, in output_to_cwl_json
output_metadata = get_metadata(galaxy_output.history_content_type, galaxy_output.history_content_id)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/planemo/galaxy/activity.py", line 575, in _get_metadata
return self._user_gi.histories.show_dataset_collection(
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/bioblend/galaxy/histories/__init__.py", line 477, in show_dataset_collection
return self._get(url=url)
File "/home/danielle/Documents/veg/tools-iuc/planemo/lib/python3.10/site-packages/bioblend/galaxy/client.py", line 162, in _get
raise ConnectionError(
bioblend.ConnectionError: HTTPConnectionPool(host='localhost', port=45995): Max retries exceeded with url: /api/histories/6edbfb9fe94476e2/contents/dataset_collections/9de04dc09b7ee1ec (Caused by NewConnectionError("HTTPConnection(host='localhost', port=45995): Failed to establish a new connection: [Errno 111] Connection refused")), 0 attempts left:
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.