elastic / elastic/beats

Unable to set max_bytes in filebeats

Open
#39,753 13 comments 0 reactions 1 assignee Claimed by @VihasMakwana View on GitHub
bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

For confirmed bugs, please report:
- Version: 8.13.45
- Operating System: ubuntu 24.04
- Discuss Forum URL: https://discuss.elastic.co/t/filebeat-send-snippet-of-xml/360364
- Steps to Reproduce: try to send a file larger than 10mb to Logstash via filebeat

filebeat.yaml
```
filebeat.inputs:
- type: filestream
id: xml-oscap
enabled: true
encoding: utf-8
#message_max_bytes: 20971520
max_bytes: 20971520
paths:
- /home/ansible/ansible_openscap/oscap-reports/*.xml
parsers:
- multiline:
type: pattern
pattern: '^<\?xml*'
#flush_pattern: '^[\S]*<\/Benchmark>'
negate: true
match: after
max_lines: 1000000000
# max_bytes: 20971520
# close_eof: true
# ============================== Filebeat modules ==============================
logging.level: debug
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644

filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml

# Set to true to enable config reloading
reload.enabled: false

# Period on which files under path should be checked for changes
#reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

# ------------------------------ Logstash Output -------------------------------
output.logstash:
hosts: ["192.168.1.220:5044"]
bulk_max_size: 20971520

# ================================= Processors =================================
processors:
#- add_host_metadata:
# when.not.contains.tags: forwarded
#- add_cloud_metadata: ~
#- add_docker_metadata: ~
#- add_kubernetes_metadata: ~
#- decode_xml:
# field: message
# target_field: TestResult
# to_lower: true
```

Error:

`
2024-05-28 16:54:38.965305489 +0000 UTC m=+6.335550826 write error: data size (11554286 bytes) is greater than the max file size (10485760 bytes)
`

It seems to filebeat not setting the max_bytes parameter I configured in filebeat.yaml

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.