aws / aws/aws-parallelcluster

HeadNode-ClustermgtdHeartbeat alarm stays in ALARM state when HeadNode OS timezone is not UTC

Open Beginner friendly
#7,576 2 comments 0 reactions 0 assignees View on GitHub
3.x Backlog Feature Request
Dominant language
Python
Stars
888
Forks
314
Avg merge
1d 10h
Merged PRs (30d)
43

Description

**Describe the issue**

When the HeadNode's OS timezone is set to a non-UTC timezone (e.g., `Asia/Tokyo / JST`), the `HeadNode-ClustermgtdHeartbeat` CloudWatch alarm constantly remains in the `ALARM` (or `INSUFFICIENT_DATA`) state.

**To Reproduce**

1. Create a cluster with AWS ParallelCluster 3.15.1.
2. Set the OS timezone of the HeadNode to `Asia/Tokyo` (JST, +09:00).
3. Check the `HeadNode-ClustermgtdHeartbeat` alarm in the CloudWatch Console. It will stay in the `ALARM` state.

**Root Cause Analysis**

The `clustermgtd.events` log outputs timestamps in UTC with a colon in the offset, for example:
`"datetime": "2026-08-18T09:30:47.592+00:00"`

However, the CloudWatch Agent configuration (`file_cloudwatch_agent_config.json`) generated by ParallelCluster defines the timestamp format without a timezone specification:
`"timestamp_format": "%Y-%m-%dT%H:%M:%S.%f"`

Because the HeadNode OS is in JST, the CloudWatch Agent parses the time `09:30:47` as JST (local time) instead of UTC. This causes the metric data to be ingested with a timestamp 9 hours in the past. Since the CloudWatch Alarm evaluates only recent data points, it fails to find them and triggers the alarm.

_Note:_ Appending `%z` to the `timestamp_format` does not work because the CloudWatch Agent only supports the `%z` format without a colon (e.g., `+0900`), and it fails to parse `+00:00`.

**Proposed Solution**

Add `"timezone": "UTC"` to the `_events` log stream definitions (e.g., `clustermgtd_events`, `slurm_resume_events`) in the CloudWatch Agent configuration template to force the agent to interpret the parsed timestamp as UTC.

```JSON
{
"log_stream_name": ".{instance_id}.clustermgtd_events",
"file_path": "/var/log/parallelcluster/clustermgtd.events",
"timestamp_format": "%Y-%m-%dT%H:%M:%S.%f",
"timezone": "UTC",
"log_group_name": "/aws/parallelcluster/..."
}
```

I have verified with AWS Support that adding `"timezone": "UTC"` resolves the issue successfully.

**Environment:**

AWS ParallelCluster Version: 3.15.1

Contributor guide

Open the contributing guide

Research direction

Start by locating the file_cloudwatch_agent_config.json template and its _events log-stream definitions, including clustermgtd_events and slurm_resume_events. Compare the generated configuration with the reported timestamp format; done means the relevant streams specify UTC and the HeadNode-ClustermgtdHeartbeat alarm receives current data on a non-UTC host.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, observability-sre
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.