Koenkk / Koenkk/zigbee2mqtt

Remote syslog: RFC5424 timestamps carry UTC values with the local offset, so log backends can silently reject every entry

Open
#33,127 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
15.7k
Forks
2k
Avg merge
18h 55m
Merged PRs (30d)
35

Description

What happened?

With advanced.log_output including syslog, the RFC5424 timestamp Zigbee2MQTT emits carries UTC values labelled with the local UTC offset. The reported instant is therefore wrong by exactly that offset.

Captured on the wire at 08:25:40 local time (06:25:40 UTC), TZ=Europe/Berlin:

<134>1 2026-09-14T06:25:40.387+02:00 homeassistant zigbee2mqtt 7 - - z2m: Configuring 'plug.livingroom.aquarium'

06:25:40+02:00 resolves to 04:25:40 UTC — two hours in the past.

This is not only a display issue. Log backends that guard their ingestion window reject such entries outright, and nothing on the Zigbee2MQTT side indicates a problem: the add-on log stays clean, the TCP connection stays established, and the messages are valid syslog. Shipping to Grafana Loki, every line was discarded:

loki_source_syslog_entries_total          354   # received and parsed by the collector
loki_source_syslog_parsing_errors_total     0   # messages are well-formed
loki_write_dropped_entries_total          171   # reason="ingester_error"
server returned HTTP status 400 Bad Request (400):
entry with timestamp 2026-09-14 04:25:40.387 +0000 UTC ignored,
reason: 'entry too far behind, oldest acceptable timestamp is: 2026-09-14T05:12:31Z'
What did you expect to happen?

A timestamp that denotes the correct instant — either UTC values with Z, or local values with the local offset.

How to reproduce it (minimal and precise)

The cause is upstream, in glossy, which winston-syslog uses to build the message. It reproduces without Zigbee2MQTT:

const glossy = require('glossy');
const p = new glossy.Produce({type: '5424', appName: 'app', pid: 7, facility: 'local0'});
const d = new Date();
console.log(process.env.TZ, d.toISOString(),
            p.produce({severity: 'info', host: 'h', date: d, message: 'probe'}));

node:22-alpine, glossy@0.1.7:

TZ Date.toISOString() (truth) glossy output error
UTC 2026-09-14T09:00:08.911Z …09:00:08.911Z… correct
Europe/Berlin 2026-09-14T09:00:08.946Z …09:00:08.946+02:00… −2 h
America/New_York 2026-09-14T09:00:08.982Z …09:00:08.982-04:00… +4 h

In glossy/lib/glossy/produce.js, generateDate() derives the offset from getTimezoneOffset() (local) while reading every component with getUTC*().

Upstream reports: squeeks/glossy#43 (open since 2019), squeeks/glossy#35 and #16 (closed without a fix); winstonjs/winston-syslog#139. glossy was last published in 2014 and winston-syslog tracks it as unmaintained in winstonjs/winston-syslog#232, so a fix arriving upstream seems unlikely.

Possible solutions

I am filing this here because Zigbee2MQTT can address it without waiting on upstream, and because the failure is invisible to users.

  1. Pass a corrected producer. winston-syslog honours options.customProducer (lib/winston-syslog.js:76: var Producer = options.customProducer || glossy.Produce;). A thin wrapper around glossy.Produce that overrides only the RFC5424 date would fix it inside lib/util/logger.ts, where the transport is already constructed. Since glossy reads the components with getUTC*(), forcing the offset to Z is sufficient.

  2. Document the TZ dependency. Per the table above, glossy is correct exactly when the process runs with TZ=UTC. A note on the logging page would save others the search. Caveat: this also moves console/file timestamps to UTC, so it is a workaround rather than a fix.

  3. At minimum, mention the limitation next to log_syslog, so users know remote timestamps are unreliable unless the container runs in UTC.

Related: #27658 reported that log_syslog.eol defaults to the literal string "/n" (forward slash) instead of a newline. It was closed as stale without a fix, and the default is still "/n" in lib/util/settings.schema.json. Anyone enabling syslog over TCP hits that one too — without eol: "\n" set explicitly, messages are not delimited and a collector merges them into a single line.

Zigbee2MQTT version

2.14.1 (Home Assistant add-on 2.14.1-1)

Adapter firmware version

EmberZNet 8.0.2 [GA]

Adapter

ember — SMLIGHT SLZB-MR4U (EFR32MG26) over TCP

Setup

Home Assistant add-on on Home Assistant Yellow (CM4), TZ=Europe/Berlin

Debug log

Not applicable — the defect is visible on the wire, see the captured message above. Zigbee2MQTT's own log shows nothing unusual, which is the point.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/util/logger.ts and inspect how the syslog transport is constructed, including the customProducer option described in the issue. Check the logging documentation and lib/util/settings.schema.json for the existing syslog configuration and eol behavior. Done means RFC5424 timestamps represent the correct instant under a non-UTC TZ, with the chosen limitation or workaround documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
backend, observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.