sincedb appears to never be created during the initial read.
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
When running logstash the sincedb files appear to only be written once the process has "caught up" with everything on disk.
Version: 5.2.2 (via https://artifacts.elastic.co/packages/5.x/apt)
Operating System: Ubuntu 16.04
Config:
```
file {
path => ["/tmp/path/*.log"]
exclude => "*.gz"
start_position => "beginning"
type => "file"
ignore_older => 86400
stat_interval => 5
sincedb_path => "/var/lib/logstash/sincedb_nginx"
sincedb_write_interval => 15
}
```
In my directories I have ~160G of log files. Ideally I would like to be able to backfill these logs into my elasticsearch cluster. The above works in that it will begin loading files of the appropriate age into kibana, however it doesn't appear to write the sincedb until it has processed all of the files.
In my current setup, running with the above config, I have a nearly hour old logstash process running that has not written any sincedb files at all. I have validated that /var/lib/logstash is owned by the logstash user, and is 755. Prior instances of logstash wrote to that directory so its not a permissions issue, and based on my past experiences it will write a sincedb once it has fully ingested all of the files that meed the above conditions.
This seems to violate the idea that the sincedb will be written ever 15 seconds as per the config.
Contributor guide
Assessment
This issue has not been assessed yet.