galaxyproject / galaxyproject/galaxy
dynamic destinations and tool input format changes
- Dominant language
- Python
- Stars
- 1.9k
- Forks
- 1.2k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 169
Description
I've been trying to work through this for much of the day, I'm perplexed. We have recently set up a dynamic destination in job_conf, and this appears to be working great. One tool is not behaving. I get the following error message:
```
Traceback (most recent call last):
File "lib/galaxy/tools/__init__.py", line 1480, in handle_single_execution
collection_info=collection_info,
File "lib/galaxy/tools/__init__.py", line 1562, in execute
return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
File "lib/galaxy/tools/actions/__init__.py", line 289, in execute
history, inp_data, inp_dataset_collections, preserved_tags, all_permissions = self._collect_inputs(tool, trans, incoming, history, current_user_roles, collection_info)
File "lib/galaxy/tools/actions/__init__.py", line 256, in _collect_inputs
inp_data, all_permissions = self._collect_input_datasets(tool, incoming, trans, history=history, current_user_roles=current_user_roles, collection_info=collection_info)
File "lib/galaxy/tools/actions/__init__.py", line 199, in _collect_input_datasets
tool.visit_inputs(param_values, visitor)
File "lib/galaxy/tools/__init__.py", line 1369, in visit_inputs
visit_input_values(self.inputs, values, callback)
File "lib/galaxy/tools/parameters/__init__.py", line 161, in visit_input_values
visit_input_values(input.cases[values['__current_case__']].inputs, values, callback, new_name_prefix, label_prefix, parent_prefix=name_prefix, **payload)
File "lib/galaxy/tools/parameters/__init__.py", line 167, in visit_input_values
callback_helper(input, input_values, name_prefix, label_prefix, parent_prefix=parent_prefix, context=context)
File "lib/galaxy/tools/parameters/__init__.py", line 130, in callback_helper
new_value = callback(**args)
File "lib/galaxy/tools/actions/__init__.py", line 141, in visitor
input_datasets[prefix + input.name] = process_dataset(value)
File "lib/galaxy/tools/actions/__init__.py", line 93, in process_dataset
data = data.get_converted_dataset(trans, target_ext, target_context=parent, history=history)
File "lib/galaxy/model/__init__.py", line 2628, in get_converted_dataset
new_dataset = next(iter(self.datatype.convert_dataset(trans, self, target_ext, return_output=True, visible=False, deps=deps, target_context=target_context, history=history).values()))
File "lib/galaxy/datatypes/data.py", line 622, in convert_dataset
converted_dataset = converter.execute(trans, incoming=params, set_output_hid=visible, history=history)[1]
File "lib/galaxy/tools/__init__.py", line 1562, in execute
return self.tool_action.execute(self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs)
File "lib/galaxy/tools/actions/__init__.py", line 350, in execute
params=wrapped_params.params,
File "lib/galaxy/tools/parameters/wrapped.py", line 44, in params
self.wrap_values(self.tool.inputs, params, skip_missing_values=not self.tool.check_values)
File "lib/galaxy/tools/parameters/wrapped.py", line 68, in wrap_values
self.wrap_values(input.cases[current].inputs, values, skip_missing_values=skip_missing_values)
File "lib/galaxy/tools/parameters/wrapped.py", line 59, in wrap_values
value = input_values[input.name]
KeyError: 'qos'
galaxy.tools.execute WARNING 2020-04-09 17:26:30,180 [p:24216,w:1,m:0] [uWSGIWorker1Core1] There was a failure executing a job for tool [galaxy2.ohsu.edu/toolshed/repos/letaw/bwa_mem/bwa_mem/0.7.9.1] - Error executing tool: 'qos'
```
"qos" is actually the name of the param in job_resource_params_conf, not the tool. Running this tool in the UI results in a red box, and the same error message. I noticed this only occurs when the inputs format needs to be transformed in to something else. For instance, when you have a fastqsanger.gz format and you need it in the uncompressed format. This is problematic for workflow invocation of the errant tool is first in line to act after the input fastqsanger.gz has been uploaded.
I don't want to speculate any further, I am usually wrong when I do that. I do realize that we can also just upgrade the tool, but we have specific constraints that mean we can't always upgrade certain workflows very quickly.
Contributor guide
Research direction
Start by reproducing the dynamic-destination case where a fastqsanger.gz input is converted before invoking the tool, then trace parameter handling through lib/galaxy/tools/parameters/wrapped.py and dataset conversion in lib/galaxy/datatypes/data.py. Confirm that the converted tool invocation no longer raises KeyError for the job_resource_params_conf qos parameter and that the UI or workflow proceeds successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100