simonsobs / simonsobs/socs

Memory leak in wiregrid encoder agent

Open
#784 2 comments 0 reactions 1 assignee View on GitHub

@sadachi5 is already working on this.

Since Oct 28, 2024.

bug needs triage
Dominant language
Python
Stars
17
Forks
20
Avg merge
1h 22m
Merged PRs (30d)
2

Description

It seems under certain conditions there is a memory leak in the wiregrid encoder agent. We ran into this on satp1 last night:
image

Right around that time the logs go from filled with "Received IRIG timeout packet" to "while calling from thread":

2024-10-24 15:39:08.363 2024-10-24T15:39:08+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:18.819 2024-10-24T15:39:18+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:28.838 2024-10-24T15:39:28+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:38.862 2024-10-24T15:39:38+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:48.894 2024-10-24T15:39:48+0000 Recieved IRIG timeout packet.
2024-10-24 15:39:58.904 2024-10-24T15:39:58+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:08.922 2024-10-24T15:40:08+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:21.938 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.939 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.940 2024-10-24T15:40:21+0000 Recieved IRIG timeout packet.
2024-10-24 15:40:21.940 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.941 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.942 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.944 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.944 2024-10-24T15:40:21+0000 while calling from thread
2024-10-24 15:40:21.946 2024-10-24T15:40:21+0000 while calling from thread

I'm not sure what's generating that "while calling from thread" message. I don't see anything with that string in it in the agent or Encoder beaglebone code, but maybe I'm missing something.

My immediate suspicion is in the nested while True: loops in grab_and_parse_data(). That's a good place to start for a deeper investigation. (And this should probably be improved, even if it doesn't end up being the problem.)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.