common-workflow-language / common-workflow-language/cwltool

tox native codegen unused variables

Open
#1,694 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
376
Forks
255
Avg merge
2d 7h
Merged PRs (30d)
12

Description

## Expected Behavior
Using the instructions in the README, developers should be able to run tox locally without errors.

## Actual Behavior
During the installation process, some autogenerated native code creates too many unused variables which crashes the entire script.

## Workflow Code
```
conda create --name cwltool_test
conda activate cwltool_test
conda install pip
pip install .
cwltool --version
pip install -rtest-requirements.txt
pip install tox; tox -p
```
That last command generates the following traceback, and then hangs...

## Full Traceback
```pytb
⠼ [2] py310-unit | py310-mypyERROR: invocation failed (exit code 1), logfile: /Users/jakefennick/cwltool/.tox/py310-mypy/log/py310-mypy-12.log
========================================================================================================================= log start ==========================================================================================================================
py310-mypy recreate: /Users/jakefennick/cwltool/.tox/py310-mypy
py310-mypy installdeps: -rrequirements.txt, -rtest-requirements.txt, -rmypy-requirements.txt
py310-mypy installed: arcp==0.2.1,argcomplete==2.0.0,attrs==21.4.0,bagit==1.8.1,CacheControl==0.12.11,certifi==2022.6.15,charset-normalizer==2.1.0,coloredlogs==15.0.1,coverage==6.4.2,execnet==1.9.0,humanfriendly==10.0,idna==3.3,iniconfig==1.1.1,isodate==0.6.1,lockfile==0.12.2,lxml==4.9.1,mistune==0.8.4,mock==4.0.3,msgpack==1.0.4,mypy==0.961,mypy-extensions==0.4.3,networkx==2.8.4,packaging==21.3,pluggy==1.0.0,prov==1.5.1,psutil==5.9.1,py==1.11.0,pydot==1.4.2,pyparsing==3.0.9,pytest==7.1.2,pytest-cov==3.0.0,pytest-forked==1.4.0,pytest-mock==3.8.2,pytest-xdist==2.5.0,python-dateutil==2.8.2,rdflib==6.1.1,requests==2.28.1,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.6,schema-salad==8.3.20220626185350,shellescape==3.8.1,six==1.16.0,tomli==2.0.1,types-mock==4.0.15,types-psutil==5.9.1,types-requests==2.28.1,types-setuptools==62.6.1,types-urllib3==1.26.16,typing_extensions==4.3.0,urllib3==1.26.10
py310-mypy run-test-pre: PYTHONHASHSEED='300611664'
py310-mypy run-test: commands[0] | make mypy mypyc PYTEST_EXTRA=
if ! test -f /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/ruamel/yaml/py.typed ; \
then \
rm -Rf typeshed/ruamel/yaml ; \
ln -s /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/ruamel/yaml \
typeshed/ruamel/ ; \
fi # if minimally required ruamel.yaml version is 0.15.99 or greater, than the above can be removed
MYPYPATH=$MYPYPATH:typeshed mypy cwltool/__init__.py cwltool/__main__.py cwltool/argparser.py cwltool/builder.py cwltool/checker.py cwltool/command_line_tool.py cwltool/context.py cwltool/cuda.py cwltool/cwlrdf.py cwltool/cwlviewer.py cwltool/docker.py cwltool/docker_id.py cwltool/env_to_stdout.py cwltool/errors.py cwltool/executors.py cwltool/expression.py cwltool/factory.py cwltool/flatten.py cwltool/job.py cwltool/load_tool.py cwltool/loghandler.py cwltool/main.py cwltool/mpi.py cwltool/mutation.py cwltool/pack.py cwltool/pathmapper.py cwltool/process.py cwltool/procgenerator.py cwltool/provenance.py cwltool/provenance_constants.py cwltool/provenance_profile.py cwltool/resolver.py cwltool/run_job.py cwltool/sandboxjs.py cwltool/secrets.py cwltool/singularity.py cwltool/singularity_utils.py cwltool/software_requirements.py cwltool/stdfsaccess.py cwltool/subgraph.py cwltool/task_queue.py cwltool/udocker.py cwltool/update.py cwltool/utils.py cwltool/validate_js.py cwltool/workflow.py cwltool/workflow_job.py tests/__init__.py tests/test_anon_types.py tests/test_check.py tests/test_conditionals.py tests/test_content_type.py tests/test_context.py tests/test_cuda.py tests/test_cwl_version.py tests/test_default_path.py tests/test_dependencies.py tests/test_docker.py tests/test_docker_info.py tests/test_docker_paths_with_colons.py tests/test_empty_input.py tests/test_environment.py tests/test_examples.py tests/test_ext.py tests/test_fetch.py tests/test_http_input.py tests/test_input_deps.py tests/test_iwdr.py tests/test_js_sandbox.py tests/test_load_tool.py tests/test_make_template.py tests/test_misc_cli.py tests/test_mpi.py tests/test_override.py tests/test_pack.py tests/test_parallel.py tests/test_path_checks.py tests/test_pathmapper.py tests/test_procgenerator.py tests/test_provenance.py tests/test_rdfprint.py tests/test_recursive_validation.py tests/test_relocate.py tests/test_schemadef.py tests/test_secrets.py tests/test_singularity.py tests/test_singularity_versions.py tests/test_stdout_stderr_log_dir.py tests/test_streaming.py tests/test_subgraph.py tests/test_target.py tests/test_tmpdir.py tests/test_toolargparse.py tests/test_trs.py tests/test_udocker.py tests/test_validate.py tests/test_validate_js.py tests/test_windows_warning.py tests/util.py
Success: no issues found in 99 source files
MYPYPATH=typeshed CWLTOOL_USE_MYPYC=1 pip install --verbose -e . \
&& pytest -rs -vv
Using pip 22.1.2 from /Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/pip (python 3.10)
Obtaining file:///Users/jakefennick/cwltool
Preparing metadata (setup.py): started
Running command python setup.py egg_info
running egg_info
creating /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info
writing /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/dependency_links.txt
writing entry points to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/entry_points.txt
writing requirements to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/requires.txt
writing top-level names to /private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/top_level.txt
writing manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
reading manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'cwltool/schemas/v1.0/salad/typeshed'
no previously-included directories found matching 'cwltool/schemas/v1.0/salad/schema_salad/tests'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/typeshed'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/schema_salad/tests'
no previously-included directories found matching 'cwltool/schemas/presentations'
no previously-included directories found matching 'cwltool/schemas/site'
no previously-included directories found matching 'cwltool/schemas/v1.0/examples'
no previously-included directories found matching 'cwltool/schemas/v1.0/v1.0'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/examples'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1'
warning: no previously-included files matching '*.py' found under directory 'cwltool/schemas'
warning: no previously-included files found matching 'debian.img'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
adding license file 'LICENSE.txt'
writing manifest file '/private/var/folders/xg/1dp6d5j960v4s6vmv680jylr0000gp/T/pip-pip-egg-info-eysfuzd7/cwltool.egg-info/SOURCES.txt'
Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: setuptools in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (62.6.0)
Requirement already satisfied: requests>=2.6.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (2.28.1)
Requirement already satisfied: ruamel.yaml<0.17.22,>=0.15 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (0.17.21)
Requirement already satisfied: rdflib<6.2.0,>=4.2.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (6.1.1)
Requirement already satisfied: shellescape<3.9,>=3.4.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (3.8.1)
Requirement already satisfied: schema-salad<9,>=8.2.20211104054942 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (8.3.20220626185350)
Requirement already satisfied: mypy-extensions in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (0.4.3)
Requirement already satisfied: psutil>=5.6.6 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (5.9.1)
Requirement already satisfied: prov==1.5.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.5.1)
Requirement already satisfied: bagit>=1.6.4 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.8.1)
Requirement already satisfied: typing-extensions in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (4.3.0)
Requirement already satisfied: coloredlogs in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (15.0.1)
Requirement already satisfied: pydot>=1.4.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (1.4.2)
Requirement already satisfied: pyparsing!=3.0.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (3.0.9)
Requirement already satisfied: argcomplete in ./.tox/py310-mypy/lib/python3.10/site-packages (from cwltool==3.1.20220714234856) (2.0.0)
Requirement already satisfied: six>=1.9.0 in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (1.16.0)
Requirement already satisfied: networkx in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (2.8.4)
Requirement already satisfied: python-dateutil in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (2.8.2)
Requirement already satisfied: lxml in ./.tox/py310-mypy/lib/python3.10/site-packages (from prov==1.5.1->cwltool==3.1.20220714234856) (4.9.1)
Requirement already satisfied: isodate in ./.tox/py310-mypy/lib/python3.10/site-packages (from rdflib<6.2.0,>=4.2.2->cwltool==3.1.20220714234856) (0.6.1)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (2.1.0)
Requirement already satisfied: idna<4,>=2.5 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (2022.6.15)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from requests>=2.6.1->cwltool==3.1.20220714234856) (1.26.10)
Requirement already satisfied: ruamel.yaml.clib>=0.2.6 in ./.tox/py310-mypy/lib/python3.10/site-packages (from ruamel.yaml<0.17.22,>=0.15->cwltool==3.1.20220714234856) (0.2.6)
Requirement already satisfied: mistune<0.9,>=0.8.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.8.4)
Requirement already satisfied: CacheControl<0.13,>=0.11.7 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.12.11)
Requirement already satisfied: lockfile>=0.9 in ./.tox/py310-mypy/lib/python3.10/site-packages (from schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (0.12.2)
Requirement already satisfied: humanfriendly>=9.1 in ./.tox/py310-mypy/lib/python3.10/site-packages (from coloredlogs->cwltool==3.1.20220714234856) (10.0)
Requirement already satisfied: msgpack>=0.5.2 in ./.tox/py310-mypy/lib/python3.10/site-packages (from CacheControl<0.13,>=0.11.7->schema-salad<9,>=8.2.20211104054942->cwltool==3.1.20220714234856) (1.0.4)
Installing collected packages: cwltool
Running setup.py develop for cwltool
Running command python setup.py develop
running develop
/Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
/Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running egg_info
writing cwltool.egg-info/PKG-INFO
writing dependency_links to cwltool.egg-info/dependency_links.txt
writing entry points to cwltool.egg-info/entry_points.txt
writing requirements to cwltool.egg-info/requires.txt
writing top-level names to cwltool.egg-info/top_level.txt
reading manifest file 'cwltool.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'cwltool/schemas/v1.0/salad/typeshed'
no previously-included directories found matching 'cwltool/schemas/v1.0/salad/schema_salad/tests'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/typeshed'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/salad/schema_salad/tests'
no previously-included directories found matching 'cwltool/schemas/presentations'
no previously-included directories found matching 'cwltool/schemas/site'
no previously-included directories found matching 'cwltool/schemas/v1.0/examples'
no previously-included directories found matching 'cwltool/schemas/v1.0/v1.0'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/examples'
no previously-included directories found matching 'cwltool/schemas/v1.1.0-dev1/v1.1.0-dev1'
warning: no previously-included files matching '*.py' found under directory 'cwltool/schemas'
warning: no previously-included files found matching 'debian.img'
warning: no previously-included files matching '*~' found anywhere in distribution
warning: no previously-included files matching '*.pyc' found anywhere in distribution
adding license file 'LICENSE.txt'
writing manifest file 'cwltool.egg-info/SOURCES.txt'
running build_ext
building '8f906987e2683b830d2d__mypyc' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/jakefennick/opt/anaconda3/envs/cwltool_test/include -fPIC -O2 -isystem /Users/jakefennick/opt/anaconda3/envs/cwltool_test/include -I/Users/jakefennick/cwltool/.tox/py310-mypy/lib/python3.10/site-packages/mypyc/lib-rt -Ibuild -I/Users/jakefennick/cwltool/.tox/py310-mypy/include -I/Users/jakefennick/opt/anaconda3/envs/cwltool_test/include/python3.10 -c build/__native_8f906987e2683b830d2d.c -o build/temp.macosx-10.9-x86_64-cpython-310/build/__native_8f906987e2683b830d2d.o -O3 -g1 -Werror -Wno-unused-function -Wno-unused-label -Wno-unreachable-code -Wno-unused-variable -Wno-unused-command-line-argument -Wno-unknown-warning-option
build/__native_8f906987e2683b830d2d.c:56380:10: error: variable 'cpy_r_r654' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r654;
^
build/__native_8f906987e2683b830d2d.c:106245:10: error: variable 'cpy_r_r278' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r278;
^
build/__native_8f906987e2683b830d2d.c:108491:10: error: variable 'cpy_r_r205' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r205;
^
build/__native_8f906987e2683b830d2d.c:108471:10: error: variable 'cpy_r_r183' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r183;
^
build/__native_8f906987e2683b830d2d.c:110989:10: error: variable 'cpy_r_r18' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r18;
^
build/__native_8f906987e2683b830d2d.c:137993:10: error: variable 'cpy_r_r520' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r520;
^
build/__native_8f906987e2683b830d2d.c:153417:10: error: variable 'cpy_r_r174' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r174;
^
build/__native_8f906987e2683b830d2d.c:153430:10: error: variable 'cpy_r_r188' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r188;
^
build/__native_8f906987e2683b830d2d.c:153403:10: error: variable 'cpy_r_r159' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r159;
^
build/__native_8f906987e2683b830d2d.c:153379:10: error: variable 'cpy_r_r133' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r133;
^
build/__native_8f906987e2683b830d2d.c:153443:10: error: variable 'cpy_r_r202' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r202;
^
build/__native_8f906987e2683b830d2d.c:174703:10: error: variable 'cpy_r_r106' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r106;
^
build/__native_8f906987e2683b830d2d.c:174710:10: error: variable 'cpy_r_r114' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r114;
^
build/__native_8f906987e2683b830d2d.c:174670:10: error: variable 'cpy_r_r76' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r76;
^
build/__native_8f906987e2683b830d2d.c:179830:10: error: variable 'cpy_r_r191' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r191;
^
build/__native_8f906987e2683b830d2d.c:194430:10: error: variable 'cpy_r_r1546' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r1546;
^
build/__native_8f906987e2683b830d2d.c:194424:10: error: variable 'cpy_r_r1539' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r1539;
^
build/__native_8f906987e2683b830d2d.c:194413:10: error: variable 'cpy_r_r1527' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r1527;
^
build/__native_8f906987e2683b830d2d.c:213579:10: error: variable 'cpy_r_r46' set but not used [-Werror,-Wunused-but-set-variable]
char cpy_r_r46;
^
fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
error: command '/usr/bin/clang' failed with exit code 1
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /Users/jakefennick/cwltool/.tox/py310-mypy/bin/python -c '
exec(compile('"'"''"'"''"'"'
# This is -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize

try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)

__file__ = %r
sys.argv[0] = __file__

if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = ""
setup_py_code = "from setuptools import setup; setup()"

exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/Users/jakefennick/cwltool/setup.py'"'"',), "", "exec"))' develop --no-deps
cwd: /Users/jakefennick/cwltool/
error: subprocess-exited-with-error

× python setup.py develop did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
make: *** [mypyc] Error 1
ERROR: InvocationError for command /usr/bin/make mypy mypyc PYTEST_EXTRA= (exited with code 2)

========================================================================================================================== log end ===========================================================================================================================
```

## Your Environment
* cwltool version: 3.1.20220714234856
Check using ``cwltool --version``

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.