since_db file loss manay file's pos
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 19h 14m
- Merged PRs (30d)
- 63
Description
Please post all product and debugging questions on our [forum](https://discuss.elastic.co/c/logstash). Your questions will reach our wider community members there, and if we confirm that there is a bug, then we can open a new issue here.
Logstash Plugins are located in a different organization: https://github.com/logstash-plugins. For bugs on specific Logstash plugins, for example, if Redis Output has a defect, please open it in the respective Redis Output repository.
For all general issues, please provide the following details for fast resolution:
- Version: logstash-2.3.3
- Operating System: CentOS 6.5
- Config File (if you have sensitive info, please remove it):
input {
file {
type => "ana_host"
path => ["/data/logstash-2.3.3/out/ana_host.out*"]
start_position => "beginning"
}
}
- Sample Data:
- Steps to Reproduce:
I have 1 file,the sincedb writed 1 line, I have 3100 files, it just writes 2383 lines to sincedb, I ava 3500 files, it just writes 3102 lines to seincedb.
**********************************
the problem is: my es doc count is ok, but when I restart logstash, the doc will repeat a lot, but not all docs. And how to fix it?
*********************************
I found the problem and I did some test, I hava 10 lines per test_file.
this is some check:
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":20,"_shards":{"total":5,"successful":5,"failed":0}}[root@master3 out]#
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
2 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":31010,"_shards":{"total":5,"successful":5,"failed":0}}
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
2383 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# for((i=3101;i<=3500;i++));do cp ana_host.out ana_host.out.$i; done
[root@master3 out]# wc -l /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
3102 /root/.sincedb_9cd44649d706eadf83b72a48116d2cba
[root@master3 out]# curl 'http://192.168.5.103:9200/_count'
{"count":35010,"_shards":{"total":5,"successful":5,"failed":0}}
Contributor guide
Assessment
This issue has not been assessed yet.