elastic / elastic/logstash

sincedb appears to never be created during the initial read.

Open
#6,900 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

When running logstash the sincedb files appear to only be written once the process has "caught up" with everything on disk.

Version: 5.2.2 (via https://artifacts.elastic.co/packages/5.x/apt)
Operating System: Ubuntu 16.04

Config:
```
file {
path => ["/tmp/path/*.log"]
exclude => "*.gz"
start_position => "beginning"
type => "file"
ignore_older => 86400
stat_interval => 5
sincedb_path => "/var/lib/logstash/sincedb_nginx"
sincedb_write_interval => 15
}
```

In my directories I have ~160G of log files. Ideally I would like to be able to backfill these logs into my elasticsearch cluster. The above works in that it will begin loading files of the appropriate age into kibana, however it doesn't appear to write the sincedb until it has processed all of the files.

In my current setup, running with the above config, I have a nearly hour old logstash process running that has not written any sincedb files at all. I have validated that /var/lib/logstash is owned by the logstash user, and is 755. Prior instances of logstash wrote to that directory so its not a permissions issue, and based on my past experiences it will write a sincedb once it has fully ingested all of the files that meed the above conditions.

This seems to violate the idea that the sincedb will be written ever 15 seconds as per the config.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.