Azure / Azure/cyclecloud-pbspro

Autoscaler is failing with KeyError: 'headnode.internal.cloudapp.net'

Open
#85 0 comments 0 reactions 2 assignees Assigned to @bwatrous View on GitHub
Dominant language
Python
Stars
16
Forks
30
Avg merge
3d 16h
Merged PRs (30d)
1

Description

On some systems, it appears that PBS commands `qmgr list sched` and `qmgr list server` sometimes return values with long names and not short names, making the research of a scheduler to failed with the following stack trace

`
2025-09-18 08:49:02,300 DEBUG: Full stacktrace
Traceback (most recent call last):
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/hpc/autoscale/clilib.py", line 1890, in main
args.func(**kwargs)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/hpc/autoscale/clilib.py", line 434, in autoscale
output_columns = output_columns or self._get_default_output_columns(config)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/hpc/autoscale/clilib.py", line 1075, in _get_default_output_columns
default_cmd = self._default_output_columns(config, cmd_name)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/pbspro/cli.py", line 76, in _default_output_columns
env = self._pbs_env(driver)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/pbspro/cli.py", line 110, in _pbs_env
self.__pbs_env = environment.from_driver(pbs_driver.config, pbs_driver)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/pbspro/environment.py", line 52, in from_driver
schedulers = pbs_driver.read_schedulers()
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/pbspro/driver.py", line 562, in read_schedulers
return read_schedulers(self.pbscmd, self.resource_definitions)
File "/opt/cycle/pbspro/venv/lib/python3.6/site-packages/pbspro/scheduler.py", line 84, in read_schedulers
server_dict = server_dicts_by_host[hostname]
KeyError: 'headnode.internal.cloudapp.net'
`

A suggested fix would be to use short names in the dictionary lookup line 83 and make sure that the scheduler is always using short names.

Image

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.