Kubernetes plugin, validate_settings crashes pod whe... | snakemake/snakemake-interface-common

This issue is related to the kubernetes plugin issue #33

I have dived into the container being ran as the pod on kubernetes and found the issue - in plugin_registry/plugin.py the function validate_settings crashes the entire pod when it receives the "settings" variable of type <class 'NoneType'>. The exact scenario where that happens is described in the above mentioned issue.

After some modifications to the function, I found out that it is in fact being called twice. Once with type "NoneType" (crashing the pod) and once with the correct class "snakemake_interface_common.plugin_registry.plugin.SettingsBase".

The fix appears to be inserting a check in the function, stopping it's further execution, for example:

validate_settings:

    def validate_settings(self, settings):
    def get_description(thefield):
        envvar = (
            f" (or environment variable {self.get_envvar(thefield.name)})"
            if thefield.metadata.get("env_var", None)
            else ""
        )
        return f"{self.get_cli_arg(thefield.name)}{envvar}"

    if settings is None:
        return None

    # rewrite for settings
    missing = [
        thefield
        for thefield in fields(settings)
        if thefield.metadata.get("required")
        and getattr(settings, thefield.name) is None
    ]
    if missing:
        cli_args = [get_description(thefield) for thefield in missing]
        raise WorkflowError(
            f"The following required arguments are missing for "
            f"plugin {self.name}: {', '.join(cli_args)}."
        )

full logs from repeated experimentation:

(base) root@3b905478bb08:/tmp/repo# /bin/sh -c "pip install --target '.snakemake/pip-deployments' snakemake-storage-plugin-fs && python -m snake make --deploy-sources snakemake-workflow-sources.bfdcb0b3ee03be3b2067e78819318ff12737df578e9d074be6bddc289acf3deb.tar.xz bfdcb0b3ee03be3b2067e78 819318ff12737df578e9d074be6bddc289acf3deb --default-storage-provider 'fs' --verbose && python -m snakemake --snakefile 'snakefile' --target-job s 'sleep:' --allowed-rules sleep --cores 8 --attempt 1 --force-use-threads --force --target-files-omit-workdir-adjustment --max-inventory-time 0 --nocolor --notemp --no-hooks --nolock --ignore-incomplete --verbose --rerun-triggers 'code' 'input' 'mtime' 'params' 'software-env' --deplo yment-method conda --conda-frontend 'conda' --shared-fs-usage 'none' --wrapper-prefix 'https://github.com/snakemake/snakemake-wrappers/raw/' --l atency-wait 5 --scheduler 'ilp' --local-storage-prefix .snakemake/storage --default-storage-provider 'fs' --default-resources base64//dG1wZGlyPX N5c3RlbV90bXBkaXI= --mode 'remote' --verbose" Collecting snakemake-storage-plugin-fs Using cached snakemake_storage_plugin_fs-1.1.1-py3-none-any.whl.metadata (1.2 kB) Collecting reretry<0.12.0,>=0.11.8 (from snakemake-storage-plugin-fs) Using cached reretry-0.11.8-py2.py3-none-any.whl.metadata (5.5 kB) Collecting snakemake-interface-common<2.0.0,>=1.17.0 (from snakemake-storage-plugin-fs) Using cached snakemake_interface_common-1.17.4-py3-none-any.whl.metadata (760 bytes) Collecting snakemake-interface-storage-plugins<5.0.0,>=4.1.0 (from snakemake-storage-plugin-fs) Using cached snakemake_interface_storage_plugins-4.2.1-py3-none-any.whl.metadata (11 kB) Collecting sysrsync<2.0.0,>=1.1.1 (from snakemake-storage-plugin-fs) Using cached sysrsync-1.1.1-py3-none-any.whl.metadata (7.2 kB) Collecting ConfigArgParse<2.0,>=1.7 (from snakemake-interface-common<2.0.0,>=1.17.0->snakemake-storage-plugin-fs) Using cached ConfigArgParse-1.7-py3-none-any.whl.metadata (23 kB) Collecting argparse-dataclass<3.0.0,>=2.0.0 (from snakemake-interface-common<2.0.0,>=1.17.0->snakemake-storage-plugin-fs) Using cached argparse_dataclass-2.0.0-py3-none-any.whl.metadata (7.2 kB) Collecting throttler>=1.2.2 (from snakemake-interface-storage-plugins<5.0.0,>=4.1.0->snakemake-storage-plugin-fs) Using cached throttler-1.2.2-py3-none-any.whl.metadata (7.4 kB) Collecting wrapt>=1.15.0 (from snakemake-interface-storage-plugins<5.0.0,>=4.1.0->snakemake-storage-plugin-fs) Using cached wrapt-1.17.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.4 kB) Collecting toml<0.11.0,>=0.10.0 (from sysrsync<2.0.0,>=1.1.1->snakemake-storage-plugin-fs) Using cached toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) Using cached snakemake_storage_plugin_fs-1.1.1-py3-none-any.whl (5.6 kB) Using cached reretry-0.11.8-py2.py3-none-any.whl (5.6 kB) Using cached snakemake_interface_common-1.17.4-py3-none-any.whl (14 kB) Using cached snakemake_interface_storage_plugins-4.2.1-py3-none-any.whl (16 kB) Using cached sysrsync-1.1.1-py3-none-any.whl (7.5 kB) Using cached argparse_dataclass-2.0.0-py3-none-any.whl (8.8 kB) Using cached ConfigArgParse-1.7-py3-none-any.whl (25 kB) Using cached throttler-1.2.2-py3-none-any.whl (7.6 kB) Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Using cached wrapt-1.17.2-cp312-cp312-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (89 kB) Installing collected packages: throttler, wrapt, toml, reretry, ConfigArgParse, argparse-dataclass, sysrsync, snakemake-interface-common, snakem ake-interface-storage-plugins, snakemake-storage-plugin-fs Successfully installed ConfigArgParse-1.7 argparse-dataclass-2.0.0 reretry-0.11.8 snakemake-interface-common-1.17.4 snakemake-interface-storage- plugins-4.2.1 snakemake-storage-plugin-fs-1.1.1 sysrsync-1.1.1 throttler-1.2.2 toml-0.10.2 wrapt-1.17.2 WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_storage_plugin_fs-1.1.1.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/reretry already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_storage_plugins-4.2.1.dist-info already exists. Specify --upg rade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_storage_plugins already exists. Specify --upgrade to force re placement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/throttler already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/argparse_dataclass.py already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/argparse_dataclass-2.0.0.dist-info already exists. Specify --upgrade to force rep lacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/sysrsync-1.1.1.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/reretry-0.11.8.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/configargparse.py already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/wrapt-1.17.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/toml-0.10.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/throttler-1.2.2.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/wrapt already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_common already exists. Specify --upgrade to force replacement . WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_interface_common-1.17.4.dist-info already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/__pycache__ already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/ConfigArgParse-1.7.dist-info already exists. Specify --upgrade to force replaceme nt. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/toml already exists. Specify --upgrade to force replacement. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/snakemake_storage_plugin_fs already exists. Specify --upgrade to force replacemen t. WARNING: Target directory /tmp/repo/.snakemake/pip-deployments/sysrsync already exists. Specify --upgrade to force replacement. WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly ren dering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-acti on option if you know what you are doing and want to suppress this warning.

------------------------------_
Validate settings

            Settings value:

None
Settings type:
Is dataclass: False
Datatype: <class 'NoneType'>

------------------------------_
Validate settings

            Settings value:

SettingsBase()
Settings type:
Is dataclass: True
Datatype: <class 'snakemake_interface_common.plugin_registry.plugin.SettingsBase'>

host: 3b905478bb08
Building DAG of jobs...
shared_storage_local_copies: False
remote_exec: True
Submitting maximum 100 job(s) over 1.0 second(s).
Using shell: /usr/bin/bash
Provided cores: 8
Rules claiming more threads will be scaled down.
Resources before job selection: {'_cores': 8, '_nodes': 9223372036854775807, '_job_count': 9223372036854775807}
Ready jobs: 1
Select jobs to execute...
Selecting jobs to run using greedy solver.
Selected jobs: 1
Resources after job selection: {'_cores': 7, '_nodes': 9223372036854775806, '_job_count': 100}
Execute 1 jobs...

[Fri Apr 4 11:29:51 2025]
localrule sleep:
jobid: 0
reason: Rules with neither input nor output files are always executed.
resources: mem_mb=20, mem_mib=20, disk_mb=1000, disk_mib=954, tmpdir=/tmp, threads=1

/opt/conda/envs/snakemake/lib/python3.12/site-packages/conda/base/context.py:198: FutureWarning: Adding 'defaults' to channel list implicitly is
deprecated and will be removed in 25.9.

To remove this warning, please choose a default channel explicitly with conda's regular configuration system, e.g. by adding 'defaults' to the l
ist of channels:

conda config --add channels defaults

For more information see https://docs.conda.io/projects/conda/en/stable/user-guide/configuration/use-condarc.html

deprecated.topic(
Waiting for more resources.
[Fri Apr 4 11:29:59 2025]
Finished jobid: 0 (Rule: sleep)
1 of 1 steps (100%) done
Storing output in storage.

I do not know whether this is the best solution, nor what effect it may have on other plugins. As such, I kindly ask to verify that this change is fit to be integrated into the repository, and if not, I'm more than eager to test more possible solutions for my problem.

Kubernetes plugin, validate_settings crashes pod when settings is <class 'NoneType'>

Description

Contributor guide

First steps

Research direction

Assessment