common-workflow-language / common-workflow-language/schema_salad

In some scenarios `fetch_text` returns empty string

Offen
#465 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
86
Forks
64
Ø Merge
19 Std. 24 Min.
Gemergte PRs (30 T.)
19

Beschreibung

This issue is related to `schema_salad.fetcher.DefaultFetcher.fetch_text`, and it is a bit difficult to reproduce. I have found it this late afternoon, when I tried next cwltool commands:

```
cwltool --debug --validate https://raw.githubusercontent.com/inab/Wetlab2Variations/eosc-life/cwl-workflows/workflows/workflow.cwl
```
```
cwltool --debug --print-deps https://raw.githubusercontent.com/inab/Wetlab2Variations/eosc-life/cwl-workflows/workflows/workflow.cwl
```

The issue is difficult to reproduce as it has appeared for me only from wireless. But I was unable to reproduce it from my workstation, for instance. When it fails the output is similar to next, but the intermediate stack trace is variable:

```
(.37) jmfernandez@olympo[10]:/tmp/ik> cwltool --debug --validate https://raw.githubusercontent.com/inab/Wetlab2Variations/eosc-life/cwl-workflows/workflows/workflow.cwl
INFO /tmp/ik/.37/bin/cwltool 3.1.20211020155521
ERROR I'm sorry, I couldn't load this CWL file.
The error was:
Traceback (most recent call last):
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/main.py", line 1067, in main
tool = make_tool(uri, loadingContext)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/load_tool.py", line 470, in make_tool
tool = loadingContext.construct_tool_object(processobj, loadingContext)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/workflow.py", line 51, in default_make_tool
return Workflow(toolpath_object, loadingContext)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/workflow.py", line 106, in __init__
step, index, loadingContext, loadingContext.prov_obj
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/workflow.py", line 150, in make_workflow_step
return WorkflowStep(toolpath_object, pos, loadingContext, parentworkflowProv)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/workflow.py", line 234, in __init__
self.embedded_tool = load_tool(toolpath_object["run"], loadingContext)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/load_tool.py", line 486, in load_tool
loadingContext, workflowobj, uri = fetch_document(argsworkflow, loadingContext)
File "/tmp/ik/.37/lib/python3.7/site-packages/cwltool/load_tool.py", line 133, in fetch_document
loadingContext.loader.fetch(fileuri, content_types=CWL_CONTENT_TYPES),
File "schema_salad/ref_resolver.py", line 985, in fetch
StopIteration
```

Digging in cwltool and later in schema_salad I found that, sometimes next block returns what it is expected, sometimes it returns an empty string:

https://github.com/common-workflow-language/schema_salad/blob/5d45ae6cdc4b9896d6e305b4372cb8aef3e42238/schema_salad/fetcher.py#L59-L75

It seems some weird behavior in `requests` library, where it does not materialize the full response of the request when the property is read (but I'm guessing, as I'm not a frequent user of such library). I guess that because I did a quick'n'dirty test, substituting `requests` library calls by `urllib.request` ones (see code below), and it started working:

```python
if scheme in ["http", "https"] and self.session is not None:
try:
headers = {}
if content_types:
headers["Accept"] = ", ".join(content_types) + ", */*;q=0.8"
req = urllib.request.Request(url, headers=headers)
#resp = self.session.get(url, headers=headers)
#resp.raise_for_status()
resp = urllib.request.urlopen(req)
except Exception as e:
raise ValidationException(f"Error fetching {url}: {e}") from e
if content_types and "content-type" in resp.headers:
content_type = resp.headers["content-type"].split(";")[:1][0]
if content_type not in content_types:
_logger.warning(
f"While fetching {url}, got content-type of "
f"'{content_type}'. Expected one of {content_types}."
)
raw_content = resp.read()
resp.close()

encoding = resp.headers.get_content_charset()
return raw_content.decode('utf-8' if encoding is None else encoding)
# return resp.text
```

Hope this helps to anyone else having this issue!

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.