BufferedTokenizer may silently drop data when oversize input has no delimiters
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
Please include the following information:
1. Logstash version (e.g. `bin/logstash --version`): `9.2.0-SNAPSHOT`
2. Logstash installation source (e.g. built from source, with a package manager: DEB/RPM, expanded from tar or zip archive, docker) _N/A_
3. How is Logstash being run (e.g. as a service/service manager: systemd, upstart, etc. Via command line, docker/kubernetes) _N/A_
**Description of the problem including expected versus actual behavior**:
In https://github.com/elastic/logstash/pull/17229 changes to the buffered tokenizer introduce an edge-case where an oversized sequence of bytes that do _not_ include a delimiter can be silently dropped.
This occurs because size validation happens _after_ the token is fully accumulated, which does not occur if no trailing separator is detected.
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
1.
2.
3.
**Provide logs (if relevant)**:
Contributor guide
Assessment
This issue has not been assessed yet.