elastic / elastic/logstash

since_db file loss manay file's pos

Open
#8,037 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.

Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.

For all general issues, please provide the following details for fast resolution:

- Version: logstash-2.3.3
- Operating System: CentOS 6.5
- Config File (if you have sensitive info, please remove it):
input {
file {
type => "ana_host"
path => ["/data/logstash-2.3.3/out/ana_host.out*"]
start_position => "beginning"
}
}
- Sample Data:
- Steps to Reproduce:
I have 1 file,the sincedb writed 1 line, I have 3100 files, it just writes 2383 lines to sincedb, I ava 3500 files, it just writes 3102 lines to seincedb.

**********************************
the problem is: my es doc count is ok, but when I restart logstash, the doc will repeat a lot, but not all docs. And how to fix it?
*********************************
I found the problem and I did some test, I hava 10 lines per test_file.
this is some check:
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":20,"_shards":{"total":5,"successful":5,"failed":0}}[root@master3 out]#
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
2 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":31010,"_shards":{"total":5,"successful":5,"failed":0}}
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
2383 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# for((i=3101;i<=3500;i++));do cp ana_host.out ana_host.out.$i; done
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
3102 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":35010,"_shards":{"total":5,"successful":5,"failed":0}}

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.