elastic / elastic/logstash

File input fails to consume globbed files

Open
#6,642 3 comments 0 reactions 0 assignees View on GitHub
unconfirmed
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Im running into a really odd issue while using the file input plugin;

Logstash 5.0.2

- 2 folders in the same path, one named 2014, the other 2015 (identical permissions at folder/file level)
- both have identical structures, 1 year worth of logs in each
- im globbing to consume a .log file as each year is split by month then day
- using start_position = beginning and sincedb_path = /dev/null, deleted residual since_db files to make sure
- 2014 is consumed and sent to ES (into the 2014 index), 2015 will not be consumed.
- Putting logstash in verbose + debug, i can see the file globbing shows all files in the 2015 folder
- I can see logstash receiving lines via debug
- Using stdin and catting the globbed 2015 dir, i can capture logs via grok and see them appear in rubydebug stdout. Its not a problem with pattern matching..
- Ive checked my config over and over.. it literally doesn't differ apart from the "2015" string in the file path and index names

Sort of at a loss as to why this is occurring... any support would be much appreciated.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.