Remote syslog: RFC5424 timestamps carry UTC values with the local offset, so log backends can silently reject every entry
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 15.7k
- Forks
- 2k
- Avg merge
- 18h 55m
- Merged PRs (30d)
- 35
Description
What happened?
With advanced.log_output including syslog, the RFC5424 timestamp Zigbee2MQTT emits carries UTC values labelled with the local UTC offset. The reported instant is therefore wrong by exactly that offset.
Captured on the wire at 08:25:40 local time (06:25:40 UTC), TZ=Europe/Berlin:
<134>1 2026-09-14T06:25:40.387+02:00 homeassistant zigbee2mqtt 7 - - z2m: Configuring 'plug.livingroom.aquarium'
06:25:40+02:00 resolves to 04:25:40 UTC — two hours in the past.
This is not only a display issue. Log backends that guard their ingestion window reject such entries outright, and nothing on the Zigbee2MQTT side indicates a problem: the add-on log stays clean, the TCP connection stays established, and the messages are valid syslog. Shipping to Grafana Loki, every line was discarded:
loki_source_syslog_entries_total 354 # received and parsed by the collector
loki_source_syslog_parsing_errors_total 0 # messages are well-formed
loki_write_dropped_entries_total 171 # reason="ingester_error"
server returned HTTP status 400 Bad Request (400):
entry with timestamp 2026-09-14 04:25:40.387 +0000 UTC ignored,
reason: 'entry too far behind, oldest acceptable timestamp is: 2026-09-14T05:12:31Z'
What did you expect to happen?
A timestamp that denotes the correct instant — either UTC values with Z, or local values with the local offset.
How to reproduce it (minimal and precise)
The cause is upstream, in glossy, which winston-syslog uses to build the message. It reproduces without Zigbee2MQTT:
const glossy = require('glossy');
const p = new glossy.Produce({type: '5424', appName: 'app', pid: 7, facility: 'local0'});
const d = new Date();
console.log(process.env.TZ, d.toISOString(),
p.produce({severity: 'info', host: 'h', date: d, message: 'probe'}));
node:22-alpine, glossy@0.1.7:
TZ |
Date.toISOString() (truth) |
glossy output | error |
|---|---|---|---|
UTC |
2026-09-14T09:00:08.911Z |
…09:00:08.911Z… |
correct |
Europe/Berlin |
2026-09-14T09:00:08.946Z |
…09:00:08.946+02:00… |
−2 h |
America/New_York |
2026-09-14T09:00:08.982Z |
…09:00:08.982-04:00… |
+4 h |
In glossy/lib/glossy/produce.js, generateDate() derives the offset from getTimezoneOffset() (local) while reading every component with getUTC*().
Upstream reports: squeeks/glossy#43 (open since 2019), squeeks/glossy#35 and #16 (closed without a fix); winstonjs/winston-syslog#139. glossy was last published in 2014 and winston-syslog tracks it as unmaintained in winstonjs/winston-syslog#232, so a fix arriving upstream seems unlikely.
Possible solutions
I am filing this here because Zigbee2MQTT can address it without waiting on upstream, and because the failure is invisible to users.
-
Pass a corrected producer.
winston-sysloghonoursoptions.customProducer(lib/winston-syslog.js:76:var Producer = options.customProducer || glossy.Produce;). A thin wrapper aroundglossy.Producethat overrides only the RFC5424 date would fix it insidelib/util/logger.ts, where the transport is already constructed. Since glossy reads the components withgetUTC*(), forcing the offset toZis sufficient. -
Document the
TZdependency. Per the table above, glossy is correct exactly when the process runs withTZ=UTC. A note on the logging page would save others the search. Caveat: this also moves console/file timestamps to UTC, so it is a workaround rather than a fix. -
At minimum, mention the limitation next to
log_syslog, so users know remote timestamps are unreliable unless the container runs in UTC.
Related: #27658 reported that log_syslog.eol defaults to the literal string "/n" (forward slash) instead of a newline. It was closed as stale without a fix, and the default is still "/n" in lib/util/settings.schema.json. Anyone enabling syslog over TCP hits that one too — without eol: "\n" set explicitly, messages are not delimited and a collector merges them into a single line.
Zigbee2MQTT version
2.14.1 (Home Assistant add-on 2.14.1-1)
Adapter firmware version
EmberZNet 8.0.2 [GA]
Adapter
ember — SMLIGHT SLZB-MR4U (EFR32MG26) over TCP
Setup
Home Assistant add-on on Home Assistant Yellow (CM4), TZ=Europe/Berlin
Debug log
Not applicable — the defect is visible on the wire, see the captured message above. Zigbee2MQTT's own log shows nothing unusual, which is the point.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/util/logger.ts and inspect how the syslog transport is constructed, including the customProducer option described in the issue. Check the logging documentation and lib/util/settings.schema.json for the existing syslog configuration and eol behavior. Done means RFC5424 timestamps represent the correct instant under a non-UTC TZ, with the chosen limitation or workaround documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100