elastic / elastic/beats

[Metricbeat][MySQL] Microsecond-precision timestamps from MySQL are incompatible with default date mapping

Open
#43,847 2 comments 0 reactions 0 assignees View on GitHub
needs_team Stalled
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

### Related Issue

- https://github.com/elastic/elasticsearch/issues/37962

> We discussed this issue internally and agreed that an error should be raised. However we are concerned by the bwc of this change and more particularly by the impact that the deprecation could have. It is allowed in 6x to send a date that contains a microseconds part by default so we'd need to log a deprecation warning in this version (and for all indices created in 6x ?) for each document that contains a deprecated format. This could slow down the indexing considerably so we agreed to document the limitation first. There is no way currently to group deprecation warning issued by different documents so we need to think of a better way to deprecate this kind of behavior.

### Situation

- Ingest source: Metricbeat → Elasticsearch
- Ingested field: `last_seen` from MySQL `performance_schema.events_statements_summary_by_digest`
- Field value example: `2025-02-24 12:01:50.406744`
- Use the default mapping provided by Metricbeat
- `name: 'last.seen' type: date`
- https://github.com/elastic/beats/blob/main/metricbeat/module/mysql/performance/_meta/fields.yml

### Problem

- Elasticsearch fails to parse date values with microsecond precision in the format:

```text
2025-02-24 12:01:50.406744
```

It throws the following error:

```text
failed to parse date field [2025-02-24 12:01:50.406744] with format [strict_date_optional_time||epoch_millis]
```

### Proposed Option

- Update the default mapping to use `date_nanos` or `text` for fields like `last.seen` to support microsecond timestamps from MySQL
- Or truncate datetime to .SSS (millisecond) precision by default
- Or document the limitation clearly and provide a workaround.

Contributor guide

Open the contributing guide

Research direction

Start with metricbeat/module/mysql/performance/_meta/fields.yml and reproduce the failure using the provided microsecond-precision last_seen value and default mapping. Compare the proposed mapping, truncation, and documentation options against Elasticsearch date parsing, then confirm the chosen behavior or clearly documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch, mysql
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.