Unable to set max_bytes in filebeats
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
For confirmed bugs, please report:
- Version: 8.13.45
- Operating System: ubuntu 24.04
- Discuss Forum URL: https://discuss.elastic.co/t/filebeat-send-snippet-of-xml/360364
- Steps to Reproduce: try to send a file larger than 10mb to Logstash via filebeat
filebeat.yaml
```
filebeat.inputs:
- type: filestream
id: xml-oscap
enabled: true
encoding: utf-8
#message_max_bytes: 20971520
max_bytes: 20971520
paths:
- /home/ansible/ansible_openscap/oscap-reports/*.xml
parsers:
- multiline:
type: pattern
pattern: '^<\?xml*'
#flush_pattern: '^[\S]*<\/Benchmark>'
negate: true
match: after
max_lines: 1000000000
# max_bytes: 20971520
# close_eof: true
# ============================== Filebeat modules ==============================
logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# ------------------------------ Logstash Output -------------------------------
output.logstash:
hosts: ["192.168.1.220:5044"]
bulk_max_size: 20971520
# ================================= Processors =================================
processors:
#- add_host_metadata:
# when.not.contains.tags: forwarded
#- add_cloud_metadata: ~
#- add_docker_metadata: ~
#- add_kubernetes_metadata: ~
#- decode_xml:
# field: message
# target_field: TestResult
# to_lower: true
```
Error:
`
2024-05-28 16:54:38.965305489 +0000 UTC m=+6.335550826 write error: data size (11554286 bytes) is greater than the max file size (10485760 bytes)
`
It seems to filebeat not setting the max_bytes parameter I configured in filebeat.yaml
Contributor guide
Assessment
This issue has not been assessed yet.