ausocean / ausocean/cloud

tv: failure state leaves camera on

Open
#535 0 comments 0 reactions 0 assignees View on GitHub
P2 P3
Dominant language
Go
Stars
1
Forks
3
Avg merge
2d 16h
Merged PRs (30d)
6

Description

The Port Noarlunga Reef stream failed but the camera was left on. This is not good and I think it could have been avoided. I'm not sure what happened, these are the logs of it starting:
```
2025-04-13 09:00:01.060 2025/04/12 23:30:01 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) (broadcast sm) got broadcast state machine; initial state: main.directIdle, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOff
2025-04-13 09:00:01.708 2025/04/12 23:30:01 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) publishing event: timeEvent
2025-04-13 09:00:01.708 2025/04/12 23:30:01 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) (broadcast sm) handling time event: 2025-04-12 23:30:01.710154737 +0000 UTC m=+439621.464457030
2025-04-13 09:00:01.708 2025/04/12 23:30:01 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) publishing event: startEvent
2025-04-13 09:00:01.708 2025/04/12 23:30:01 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) (broadcast sm) handling start event
2025-04-13 09:00:02.346 2025/04/12 23:30:02 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) (broadcast sm) transitioning from main.directIdle to main.directStarting
2025-04-13 09:00:05.449 2025/04/12 23:30:05 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) could not get today's broadcast start time: error parsing time: parsing time "" as "2006-01-02T15:04:05Z07:00": cannot parse "" as "2006"
2025-04-13 09:00:08.705 2025/04/12 23:30:08 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) inserting broadcast, name: Port Noarlunga Reef 13/04/2025, privacy: unlisted, start: 2025-04-12 23:31:05.649739225 +0000 UTC m=+439685.404041517, end: 2025-04-13 09:00:00 +0000 UTC
2025-04-13 09:00:10.667 2025/04/12 23:30:10 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) Broadcast "62cY5XS1d1w" with title "Port Noarlunga Reef 13/04/2025" was published at 2025-04-12T23:30:09Z.
2025-04-13 09:00:10.667 2025/04/12 23:30:10 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) setting category to "Science & Technology" for ID: 62cY5XS1d1w
2025-04-13 09:00:12.509 2025/04/12 23:30:12 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) set category response: {200 map[Alt-Svc:[h3=":443"; ma=2592000,h3-29=":443"; ma=2592000] Content-Type:[application/json; charset=UTF-8] Date:[Sat, 12 Apr 2025 23:30:12 GMT] Server:[scaffolding on HTTPServer2] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]}
2025-04-13 09:00:12.509 2025/04/12 23:30:12 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) inserting stream, name: Port Noarlunga Reef, res: 1080p, typ: rtmp, rate: 30fps
2025-04-13 09:00:13.486 2025/04/12 23:30:13 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) Stream "tnxOgHYhiDCCB0bFP94KKg1744500612935269" with title "Port Noarlunga Reef" was inserted.
2025-04-13 09:00:14.410 2025/04/12 23:30:14 (name: Port Noarlunga Reef, id: 9uSXRVhxwQ8) Broadcast "62cY5XS1d1w" was bound to stream "tnxOgHYhiDCCB0bFP94KKg1744500612935269".
2025-04-13 09:00:15.661 2025/04/12 23:30:15 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: hardwareStartRequestEvent
2025-04-13 09:00:16.292 2025/04/12 23:30:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling hardware start request event
2025-04-13 09:00:16.292 2025/04/12 23:30:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOff
2025-04-13 09:00:16.939 2025/04/12 23:30:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) transitioning from main.hardwareOff to main.hardwareStarting
2025-04-13 09:00:17.673 2025/04/12 23:30:17 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) got AlarmVoltage for 8813bf047fd8: 779
2025-04-13 09:00:18.273 2025/04/12 23:30:18 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) checking 8813bf047fd8.Power2 variable exists
2025-04-13 09:00:18.472 2025/04/12 23:30:18 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) 8813bf047fd8.Power2 variable exists, setting to value: true
2025-04-13 09:00:18.921 2025/04/12 23:30:18 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) checking a0a0a0b694fc.mode variable exists
2025-04-13 09:00:19.121 2025/04/12 23:30:19 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) a0a0a0b694fc.mode variable exists, setting to value: Normal
2025-04-13 09:00:19.594 2025/04/12 23:30:19 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) checking a0a0a0b694fc.RTMPURL variable exists
2025-04-13 09:00:19.794 2025/04/12 23:30:19 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) a0a0a0b694fc.RTMPURL variable exists, setting to value: rtmp://a.rtmp.youtube.com/live2/4q0v-ba5v-8qd8-9cgs-dmg6
2025-04-13 09:00:20.246 2025/04/12 23:30:20 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareStarting
2025-04-13 09:00:21.534 2025/04/12 23:30:21 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareStarting
2025-04-13 09:00:22.812 2025/04/12 23:30:22 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:00:22.812 2025/04/12 23:30:22 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) checking status of device with mac: 176611751531772
2025-04-13 09:00:23.213 2025/04/12 23:30:23 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) status from DeviceIsUp check: false
2025-04-13 09:00:23.213 2025/04/12 23:30:23 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareStarting
2025-04-13 09:01:01.074 2025/04/12 23:31:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareStarting
2025-04-13 09:01:01.707 2025/04/12 23:31:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:01:01.707 2025/04/12 23:31:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:31:01.707965445 +0000 UTC m=+439741.347956729
2025-04-13 09:01:02.339 2025/04/12 23:31:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:01:02.339 2025/04/12 23:31:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) checking status of device with mac: 176611751531772
2025-04-13 09:01:02.736 2025/04/12 23:31:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) status from DeviceIsUp check: false
2025-04-13 09:01:02.736 2025/04/12 23:31:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareStarting
2025-04-13 09:02:01.084 2025/04/12 23:32:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareStarting
2025-04-13 09:02:01.733 2025/04/12 23:32:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:02:01.733 2025/04/12 23:32:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:32:01.735549735 +0000 UTC m=+439801.372141171
2025-04-13 09:02:02.421 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:02:02.421 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) checking status of device with mac: 176611751531772
2025-04-13 09:02:02.826 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) status from DeviceIsUp check: true
2025-04-13 09:02:02.826 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: hardwareStartedEvent
2025-04-13 09:02:02.826 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling hardware started event
2025-04-13 09:02:02.826 2025/04/12 23:32:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) starting youtube broadcast object
2025-04-13 09:02:03.463 2025/04/12 23:32:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling hardware started event
2025-04-13 09:02:03.463 2025/04/12 23:32:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareStarting
2025-04-13 09:02:04.151 2025/04/12 23:32:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) transitioning from main.hardwareStarting to main.hardwareOn
2025-04-13 09:02:04.151 2025/04/12 23:32:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:02:04.791 2025/04/12 23:32:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:02:04.846 2025/04/12 23:32:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) waiting for ready status...
2025-04-13 09:02:20.688 2025/04/12 23:32:20 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) status ready reached, breaking...
2025-04-13 09:02:20.688 2025/04/12 23:32:20 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) waiting for active status...
2025-04-13 09:02:36.218 2025/04/12 23:32:36 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) status active reached, breaking...
2025-04-13 09:02:36.218 2025/04/12 23:32:36 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) ID: 62cY5XS1d1w, requesting transition to testing status...
2025-04-13 09:02:38.159 2025/04/12 23:32:38 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) waiting for testing status...
2025-04-13 09:02:53.626 2025/04/12 23:32:53 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) status testing reached, breaking...
2025-04-13 09:02:53.626 2025/04/12 23:32:53 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) ID: 62cY5XS1d1w, requesting transition to live status...
2025-04-13 09:02:55.644 2025/04/12 23:32:55 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) waiting for live status...
2025-04-13 09:03:01.060 2025/04/12 23:33:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOn
2025-04-13 09:03:01.693 2025/04/12 23:33:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:03:01.693 2025/04/12 23:33:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:33:01.695228367 +0000 UTC m=+439861.331819804
2025-04-13 09:03:02.340 2025/04/12 23:33:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:03:02.340 2025/04/12 23:33:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:03:11.154 2025/04/12 23:33:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) status live reached, breaking...
2025-04-13 09:03:11.613 2025/04/12 23:33:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: startedEvent
2025-04-13 09:03:11.613 2025/04/12 23:33:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) storing event after cancel: startedEvent
2025-04-13 09:04:01.051 2025/04/12 23:34:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [startedEvent], HardwareState: hardwareOn
2025-04-13 09:04:01.051 2025/04/12 23:34:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing stored event: startedEvent
2025-04-13 09:04:01.051 2025/04/12 23:34:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: startedEvent
2025-04-13 09:04:01.051 2025/04/12 23:34:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling started event
2025-04-13 09:04:01.793 2025/04/12 23:34:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) transitioning from main.directStarting to main.directLive
2025-04-13 09:04:02.441 2025/04/12 23:34:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:04:03.736 2025/04/12 23:34:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:04:03.736 2025/04/12 23:34:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:34:03.737369892 +0000 UTC m=+439923.373961328
2025-04-13 09:04:03.736 2025/04/12 23:34:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:04:03.736 2025/04/12 23:34:03 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:04:07.104 2025/04/12 23:34:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) issue found: noData
2025-04-13 09:04:10.163 2025/04/12 23:34:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:04:11.518 2025/04/12 23:34:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: statusCheckDueEvent
2025-04-13 09:04:11.518 2025/04/12 23:34:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling status check
2025-04-13 09:04:15.429 2025/04/12 23:34:15 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:04:17.399 2025/04/12 23:34:17 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: chatMessageDueEvent
2025-04-13 09:04:17.399 2025/04/12 23:34:17 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) ignoring sensors
2025-04-13 09:04:18.741 2025/04/12 23:34:18 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:04:22.094 2025/04/12 23:34:22 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:04:22.094 2025/04/12 23:34:22 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:05:04.395 2025/04/12 23:35:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLive, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOn
2025-04-13 09:05:05.402 2025/04/12 23:35:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:05:05.402 2025/04/12 23:35:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:35:05.403639855 +0000 UTC m=+439985.040231291
2025-04-13 09:05:05.402 2025/04/12 23:35:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:05:05.402 2025/04/12 23:35:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:05:08.665 2025/04/12 23:35:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: goodHealthEvent
2025-04-13 09:05:08.665 2025/04/12 23:35:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling good health event
2025-04-13 09:05:09.481 2025/04/12 23:35:09 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:05:11.207 2025/04/12 23:35:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:05:11.969 2025/04/12 23:35:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: statusCheckDueEvent
2025-04-13 09:05:11.969 2025/04/12 23:35:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling status check
2025-04-13 09:05:15.465 2025/04/12 23:35:15 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:05:16.765 2025/04/12 23:35:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:05:16.765 2025/04/12 23:35:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:06:01.183 2025/04/12 23:36:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLive, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOn
2025-04-13 09:06:01.818 2025/04/12 23:36:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:06:01.818 2025/04/12 23:36:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:36:01.819317973 +0000 UTC m=+440041.459309266
2025-04-13 09:06:02.468 2025/04/12 23:36:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:06:02.468 2025/04/12 23:36:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:07:01.135 2025/04/12 23:37:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLive, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOn
2025-04-13 09:07:01.802 2025/04/12 23:37:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:07:01.802 2025/04/12 23:37:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:37:01.803716193 +0000 UTC m=+440101.440307629
2025-04-13 09:07:01.802 2025/04/12 23:37:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:07:01.802 2025/04/12 23:37:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:07:04.698 2025/04/12 23:37:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: goodHealthEvent
2025-04-13 09:07:04.698 2025/04/12 23:37:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling good health event

```

Then the logs for failing:
```
2025-04-13 09:19:01.732 2025/04/12 23:49:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:19:01.732 2025/04/12 23:49:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:19:05.135 2025/04/12 23:49:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) issue found: noData
2025-04-13 09:19:05.135 2025/04/12 23:49:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: badHealthEvent
2025-04-13 09:19:05.135 2025/04/12 23:49:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling bad health event
2025-04-13 09:19:05.786 2025/04/12 23:49:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) transitioning from main.directLive to main.directLiveUnhealthy
2025-04-13 09:19:06.416 2025/04/12 23:49:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:19:07.068 2025/04/12 23:49:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) poor stream health, status: noData
2025-04-13 09:19:07.068 2025/04/12 23:49:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) broadcast context notifier is nil, setting to global notifier
2025-04-13 09:19:07.278 2025/04/12 23:49:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) could not send health notification: no recipient
2025-04-13 09:19:08.556 2025/04/12 23:49:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:19:09.241 2025/04/12 23:49:09 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: statusCheckDueEvent
2025-04-13 09:19:09.241 2025/04/12 23:49:09 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling status check
2025-04-13 09:19:13.705 2025/04/12 23:49:13 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:19:14.423 2025/04/12 23:49:14 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: chatMessageDueEvent
2025-04-13 09:19:14.423 2025/04/12 23:49:14 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) ignoring sensors
2025-04-13 09:19:15.084 2025/04/12 23:49:15 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:19:16.361 2025/04/12 23:49:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:19:16.361 2025/04/12 23:49:16 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:20:01.083 2025/04/12 23:50:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLiveUnhealthy, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareOn
2025-04-13 09:20:01.723 2025/04/12 23:50:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:20:01.723 2025/04/12 23:50:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:50:01.724643393 +0000 UTC m=+440881.364634676
2025-04-13 09:20:01.723 2025/04/12 23:50:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:20:01.723 2025/04/12 23:50:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:20:05.132 2025/04/12 23:50:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) issue found: noData
2025-04-13 09:20:06.412 2025/04/12 23:50:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:20:07.050 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) state main.directLiveUnhealthy is fixable, trying to fix
2025-04-13 09:20:07.050 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) attempting to fix broadcast by hardware restart request (attempts: 1, max attempts: 3)
2025-04-13 09:20:07.050 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) broadcast context notifier is nil, setting to global notifier
2025-04-13 09:20:07.249 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) could not send health notification: no recipient
2025-04-13 09:20:07.249 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: hardwareResetRequestEvent
2025-04-13 09:20:07.893 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling hardware reset request event
2025-04-13 09:20:07.893 2025/04/12 23:50:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareOn
2025-04-13 09:20:08.525 2025/04/12 23:50:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) transitioning from main.hardwareOn to main.hardwareRestarting
2025-04-13 09:20:08.525 2025/04/12 23:50:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) checking a0a0a0b694fc.mode variable exists

2025-04-13 09:20:08.724 2025/04/12 23:50:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) a0a0a0b694fc.mode variable exists, setting to value: shutdown
2025-04-13 09:20:09.181 2025/04/12 23:50:09 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareRestarting
2025-04-13 09:20:10.476 2025/04/12 23:50:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:20:10.878 2025/04/12 23:50:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareRestarting
2025-04-13 09:21:01.158 2025/04/12 23:51:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLiveUnhealthy, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareRestarting
2025-04-13 09:21:01.789 2025/04/12 23:51:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:21:01.789 2025/04/12 23:51:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:51:01.790049391 +0000 UTC m=+440941.430040675
2025-04-13 09:21:01.789 2025/04/12 23:51:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: healthCheckDueEvent
2025-04-13 09:21:01.789 2025/04/12 23:51:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling health check
2025-04-13 09:21:04.654 2025/04/12 23:51:04 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) issue found: noData
2025-04-13 09:21:05.936 2025/04/12 23:51:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareRestarting
2025-04-13 09:21:06.583 2025/04/12 23:51:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: statusCheckDueEvent
2025-04-13 09:21:06.583 2025/04/12 23:51:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling status check
2025-04-13 09:21:10.519 2025/04/12 23:51:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareRestarting
2025-04-13 09:21:11.182 2025/04/12 23:51:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) state main.directLiveUnhealthy is fixable, trying to fix
2025-04-13 09:21:11.821 2025/04/12 23:51:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:21:12.215 2025/04/12 23:51:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareRestarting
2025-04-13 09:22:01.088 2025/04/12 23:52:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directLiveUnhealthy, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareRestarting
2025-04-13 09:22:01.720 2025/04/12 23:52:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:22:01.720 2025/04/12 23:52:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-12 23:52:01.721577289 +0000 UTC m=+441001.358168725
2025-04-13 09:22:01.720 2025/04/12 23:52:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) state main.directLiveUnhealthy is fixable, trying to fix
2025-04-13 09:22:02.369 2025/04/12 23:52:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
```
Then it tried to start again for some reason:
```
2025-04-13 09:36:06.366 2025/04/13 00:06:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) poor stream health, status: noData
2025-04-13 09:36:06.366 2025/04/13 00:06:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) broadcast context notifier is nil, setting to global notifier
2025-04-13 09:36:06.560 2025/04/13 00:06:06 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) could not send health notification: no recipient
2025-04-13 09:36:07.827 2025/04/13 00:06:07 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:36:08.476 2025/04/13 00:06:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: statusCheckDueEvent
2025-04-13 09:36:08.476 2025/04/13 00:06:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) handling status check
2025-04-13 09:36:11.809 2025/04/13 00:06:11 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:36:12.445 2025/04/13 00:06:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) state main.directLiveUnhealthy is fixable, trying to fix
2025-04-13 09:36:12.445 2025/04/13 00:06:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) failed to fix broadcast, requesting broadcast finish (attempts: 4, max attempts: 3)
2025-04-13 09:36:12.641 2025/04/13 00:06:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) could not send health notification: no recipient
2025-04-13 09:36:12.641 2025/04/13 00:06:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: finishEvent
2025-04-13 09:36:12.641 2025/04/13 00:06:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling finish event
2025-04-13 09:36:13.278 2025/04/13 00:06:13 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) transitioning from main.directLiveUnhealthy to main.directIdle
2025-04-13 09:36:13.278 2025/04/13 00:06:13 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) stopping
2025-04-13 09:36:18.680 2025/04/13 00:06:18 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) ID: 62cY5XS1d1w, requesting transition to complete status...
2025-04-13 09:36:20.689 2025/04/13 00:06:20 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) saving, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:36:24.156 2025/04/13 00:06:24 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: hardwareStopRequestEvent
2025-04-13 09:36:24.790 2025/04/13 00:06:24 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling hardware stop request event

2025-04-13 09:36:24.790 2025/04/13 00:06:24 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) unexpected event hardwareStopRequestEvent in current state main.hardwareFailure
2025-04-13 09:36:24.790 2025/04/13 00:06:24 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:36:26.077 2025/04/13 00:06:26 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:36:27.369 2025/04/13 00:06:27 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) handling time event
2025-04-13 09:36:27.369 2025/04/13 00:06:27 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:37:01.091 2025/04/13 00:07:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) got broadcast state machine; initial state: main.directIdle, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:37:01.718 2025/04/13 00:07:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: timeEvent
2025-04-13 09:37:01.718 2025/04/13 00:07:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling time event: 2025-04-13 00:07:01.719647026 +0000 UTC m=+441901.356238462
2025-04-13 09:37:01.718 2025/04/13 00:07:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) publishing event: startEvent
2025-04-13 09:37:01.718 2025/04/13 00:07:01 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) handling start event
2025-04-13 09:37:02.357 2025/04/13 00:07:02 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) (broadcast sm) transitioning from main.directIdle to main.directStarting
2025-04-13 09:37:05.682 2025/04/13 00:07:05 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) broadcast does not exist for today, last start time: 2025-04-12 23:31:05 +0000 UTC
2025-04-13 09:37:08.772 2025/04/13 00:07:08 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) inserting broadcast, name: Port Noarlunga Reef 13/04/2025, privacy: unlisted, start: 2025-04-13 00:08:05.882601352 +0000 UTC m=+441965.519192787, end: 2025-04-13 09:00:00 +0000 UTC
2025-04-13 09:37:10.821 2025/04/13 00:07:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) Broadcast "yKSDnE8Rc0I" with title "Port Noarlunga Reef 13/04/2025" was published at 2025-04-13T00:07:09Z.
2025-04-13 09:37:10.821 2025/04/13 00:07:10 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) setting category to "Science & Technology" for ID: yKSDnE8Rc0I
2025-04-13 09:37:12.924 2025/04/13 00:07:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) set category response: {200 map[Alt-Svc:[h3=":443"; ma=2592000,h3-29=":443"; ma=2592000] Content-Type:[application/json; charset=UTF-8] Date:[Sun, 13 Apr 2025 00:07:12 GMT] Server:[scaffolding on HTTPServer2] Vary:[Origin X-Origin Referer] X-Content-Type-Options:[nosniff] X-Frame-Options:[SAMEORIGIN] X-Xss-Protection:[0]]}
2025-04-13 09:37:12.924 2025/04/13 00:07:12 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) inserting stream, name: Port Noarlunga Reef, res: 1080p, typ: rtmp, rate: 30fps
2025-04-13 09:37:13.935 2025/04/13 00:07:13 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) Stream "tnxOgHYhiDCCB0bFP94KKg1744502833334134" with title "Port Noarlunga Reef" was inserted.
2025-04-13 09:37:15.094 2025/04/13 00:07:15 (name: Port Noarlunga Reef, id: 62cY5XS1d1w) Broadcast "yKSDnE8Rc0I" was bound to stream "tnxOgHYhiDCCB0bFP94KKg1744502833334134".
2025-04-13 09:37:16.309 2025/04/13 00:07:16 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: hardwareStartRequestEvent
2025-04-13 09:37:16.952 2025/04/13 00:07:16 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling hardware start request event
2025-04-13 09:37:16.952 2025/04/13 00:07:16 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) unexpected event hardwareStartRequestEvent in current state main.hardwareFailure
2025-04-13 09:37:16.952 2025/04/13 00:07:16 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:37:18.215 2025/04/13 00:07:18 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:37:19.467 2025/04/13 00:07:19 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling time event
2025-04-13 09:37:19.467 2025/04/13 00:07:19 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:38:01.069 2025/04/13 00:08:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:38:01.787 2025/04/13 00:08:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: timeEvent
2025-04-13 09:38:01.787 2025/04/13 00:08:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) handling time event: 2025-04-13 00:08:01.788290361 +0000 UTC m=+441961.428281644
2025-04-13 09:38:02.421 2025/04/13 00:08:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling time event
2025-04-13 09:38:02.421 2025/04/13 00:08:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:39:01.101 2025/04/13 00:09:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:39:01.731 2025/04/13 00:09:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: timeEvent
2025-04-13 09:39:01.731 2025/04/13 00:09:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) handling time event: 2025-04-13 00:09:01.732654835 +0000 UTC m=+442021.372646126
2025-04-13 09:39:02.362 2025/04/13 00:09:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling time event
2025-04-13 09:39:02.362 2025/04/13 00:09:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
```
Then it finally gave up:
```
2025-04-13 09:56:02.325 2025/04/13 00:26:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:57:01.134 2025/04/13 00:27:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) got broadcast state machine; initial state: main.directStarting, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:57:01.779 2025/04/13 00:27:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: timeEvent
2025-04-13 09:57:01.779 2025/04/13 00:27:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) handling time event: 2025-04-13 00:27:01.780833202 +0000 UTC m=+443101.420824495
2025-04-13 09:57:01.779 2025/04/13 00:27:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) timed out, last entered: 2025-04-13 00:21:02.382817268 +0000 UTC, time now: 2025-04-13 00:27:01.780833202 +0000 UTC m=+443101.420824495
2025-04-13 09:57:01.779 2025/04/13 00:27:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) failed to start broadcast: direct starting timed out
2025-04-13 09:57:02.173 2025/04/13 00:27:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) broadcast start failure limit reached after 3 attempts, entering broadcast failure state, error: direct starting timed out)
2025-04-13 09:57:02.173 2025/04/13 00:27:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) broadcast context notifier is nil, setting to global notifier
2025-04-13 09:57:02.374 2025/04/13 00:27:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) could not send health notification: no recipient
2025-04-13 09:57:02.613 2025/04/13 00:27:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: criticalFailureEvent
2025-04-13 09:57:02.613 2025/04/13 00:27:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) handling critical failure event
2025-04-13 09:57:03.264 2025/04/13 00:27:03 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) transitioning from main.directStarting to main.directFailure
2025-04-13 09:57:03.264 2025/04/13 00:27:03 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) stopping
2025-04-13 09:57:08.142 2025/04/13 00:27:08 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) ID: yKSDnE8Rc0I, requesting transition to complete status...
2025-04-13 09:57:08.762 2025/04/13 00:27:08 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) could not stop broadcast on direct failure entry: could not complete broadcast: complete broadcast error: googleapi: Error 403: Invalid transition, invalidTransition
2025-04-13 09:57:08.762 2025/04/13 00:27:08 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: hardwareStopRequestEvent
2025-04-13 09:57:09.405 2025/04/13 00:27:09 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling hardware stop request event

2025-04-13 09:57:09.405 2025/04/13 00:27:09 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) unexpected event hardwareStopRequestEvent in current state main.hardwareFailure
2025-04-13 09:57:09.405 2025/04/13 00:27:09 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:57:10.677 2025/04/13 00:27:10 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:57:11.956 2025/04/13 00:27:11 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling time event
2025-04-13 09:57:11.956 2025/04/13 00:27:11 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
2025-04-13 09:58:01.059 2025/04/13 00:28:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) got broadcast state machine; initial state: main.directFailure, start: 2025-04-12 23:30:00 +0000 UTC, end: 2025-04-13 09:00:00 +0000 UTC, cfg: (config logging disabled) Events: [], HardwareState: hardwareFailure
2025-04-13 09:58:01.705 2025/04/13 00:28:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) publishing event: timeEvent
2025-04-13 09:58:01.705 2025/04/13 00:28:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) handling time event: 2025-04-13 00:28:01.706135705 +0000 UTC m=+443161.346126989
2025-04-13 09:58:01.705 2025/04/13 00:28:01 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (broadcast sm) unexpected event timeEvent in current state main.directFailure
2025-04-13 09:58:02.347 2025/04/13 00:28:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) handling time event
2025-04-13 09:58:02.347 2025/04/13 00:28:02 (name: Port Noarlunga Reef, id: yKSDnE8Rc0I) (hardware sm) saving hardware state to config: hardwareFailure
```

What's weird is that the device is up check stops after 9:02, two minutes after starting.. I thought this is usually checked more often, especially during restarts.

This needs more investigation.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the broadcast and hardware state-machine transitions shown in the logs, especially the failure path after hardware reaches hardwareOn. Reproduce or test a failed stream startup and verify that the failure path leaves the camera hardware off.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.