Supervisor PID client is None after outage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 17
- Forks
- 20
- Avg merge
- 1h 22m
- Merged PRs (30d)
- 2
Description
We saw some very strange HWP behavior today which is likely related to an outage.
The supervisor seems unable to connect to the PID client, with logs like:
Completed with state: ControlState.Error(traceback='Traceback (most recent call last):\n File "/opt/venv/lib/python3.10/site-packages/socs/agents/hwp_supervisor/agent.py", line 1087, in update\n self.run_and_validate(clients.pid.declare_freq,\nAttributeError: \'NoneType\' object has no attribute \'declare_freq\'\n', start_time=1729869143.9402528)
After the outage, Bryce says they manually restarted the main process, but not the entire supervisor, putting it in a weird state. The supervisor should be robust to this, and should be sure not to attempt to run any operations if it cannot communicate to all necessary agents.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read /opt/venv/lib/python3.10/site-packages/socs/agents/hwp_supervisor/agent.py around update at line 1087, then trace how clients.pid is initialized after the main process restarts. Review the linked Grafana logs for the outage behavior. Done means the supervisor avoids operations when required agents are unavailable instead of dereferencing a missing PID client.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100