HeadNode-ClustermgtdHeartbeat alarm stays in ALARM state when HeadNode OS timezone is not UTC
- Dominant language
- Python
- Stars
- 888
- Forks
- 314
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 43
Description
**Describe the issue**
When the HeadNode's OS timezone is set to a non-UTC timezone (e.g., `Asia/Tokyo / JST`), the `HeadNode-ClustermgtdHeartbeat` CloudWatch alarm constantly remains in the `ALARM` (or `INSUFFICIENT_DATA`) state.
**To Reproduce**
1. Create a cluster with AWS ParallelCluster 3.15.1.
2. Set the OS timezone of the HeadNode to `Asia/Tokyo` (JST, +09:00).
3. Check the `HeadNode-ClustermgtdHeartbeat` alarm in the CloudWatch Console. It will stay in the `ALARM` state.
**Root Cause Analysis**
The `clustermgtd.events` log outputs timestamps in UTC with a colon in the offset, for example:
`"datetime": "2026-08-18T09:30:47.592+00:00"`
However, the CloudWatch Agent configuration (`file_cloudwatch_agent_config.json`) generated by ParallelCluster defines the timestamp format without a timezone specification:
`"timestamp_format": "%Y-%m-%dT%H:%M:%S.%f"`
Because the HeadNode OS is in JST, the CloudWatch Agent parses the time `09:30:47` as JST (local time) instead of UTC. This causes the metric data to be ingested with a timestamp 9 hours in the past. Since the CloudWatch Alarm evaluates only recent data points, it fails to find them and triggers the alarm.
_Note:_ Appending `%z` to the `timestamp_format` does not work because the CloudWatch Agent only supports the `%z` format without a colon (e.g., `+0900`), and it fails to parse `+00:00`.
**Proposed Solution**
Add `"timezone": "UTC"` to the `_events` log stream definitions (e.g., `clustermgtd_events`, `slurm_resume_events`) in the CloudWatch Agent configuration template to force the agent to interpret the parsed timestamp as UTC.
```JSON
{
"log_stream_name": ".{instance_id}.clustermgtd_events",
"file_path": "/var/log/parallelcluster/clustermgtd.events",
"timestamp_format": "%Y-%m-%dT%H:%M:%S.%f",
"timezone": "UTC",
"log_group_name": "/aws/parallelcluster/..."
}
```
I have verified with AWS Support that adding `"timezone": "UTC"` resolves the issue successfully.
**Environment:**
AWS ParallelCluster Version: 3.15.1
Contributor guide
Research direction
Start by locating the file_cloudwatch_agent_config.json template and its _events log-stream definitions, including clustermgtd_events and slurm_resume_events. Compare the generated configuration with the reported timestamp format; done means the relevant streams specify UTC and the HeadNode-ClustermgtdHeartbeat alarm receives current data on a non-UTC host.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100