influxdata / influxdata/influxdb

Points beyond retention policy dropped for infinite retention policy

Open
#15,050 13 comments 0 reactions 1 assignee Claimed by @gwossum View on GitHub
1.x needs-backport/1.7 needs-backport/1.8
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
I'm writing data to InfluxDB in bulk using python library. If I push data for some specific period it's giving me error saying that `400: {"error":"partial write: points beyond retention policy dropped=1"}`

I did little dig-up to find out why it's giving error for this specific period and found that some shards are overlapping. Can it be the reason?

I tried writing data one at a time and it's working for the same duration same data.

```
$ show shards
name: mydatabase
id database retention_policy shard_group start_time end_time expiry_time owners
-- -------- ---------------- ----------- ---------- -------- ----------- ------
447 mydatabase autogen 447 2018-10-22T00:00:00Z 2018-11-19T00:00:00Z 2018-11-19T00:00:00Z
321 mydatabase autogen 321 2018-11-12T00:00:00Z 2018-11-19T00:00:00Z 2018-11-19T00:00:00Z
449 mydatabase autogen 449 2018-11-19T00:00:00Z 2018-12-17T00:00:00Z 2018-12-17T00:00:00Z
67 mydatabase autogen 67 2018-12-24T00:00:00Z 2018-12-31T00:00:00Z 2018-12-31T00:00:00Z
3 mydatabase autogen 3 2018-12-31T00:00:00Z 2019-01-07T00:00:00Z 2019-01-07T00:00:00Z
453 mydatabase autogen 453 2018-12-17T00:00:00Z 2019-01-14T00:00:00Z 2019-01-14T00:00:00Z
4 mydatabase autogen 4 2019-01-07T00:00:00Z 2019-01-14T00:00:00Z 2019-01-14T00:00:00Z
6 mydatabase autogen 6 2019-01-14T00:00:00Z 2019-01-21T00:00:00Z 2019-01-21T00:00:00Z
8 mydatabase autogen 8 2019-01-21T00:00:00Z 2019-01-28T00:00:00Z 2019-01-28T00:00:00Z
10 mydatabase autogen 10 2019-01-28T00:00:00Z 2019-02-04T00:00:00Z 2019-02-04T00:00:00Z
```
Also check retention policy which is set to infinite.
```
> show retention policies
name duration shardGroupDuration replicaN default
---- -------- ------------------ -------- -------
autogen 0s 672h0m0s 1 true
```
I did changed shardGroupDuration multiple time. Can it cause issue?

__Expected behavior:__
Data should have been written in chunk as well.

__Actual behavior:__
Data is dropped by retention policy

__Environment info:__

* System info: Linux 4.4.0-1087-aws x86_64
* InfluxDB version: InfluxDB v1.7.6 (git: 1.7 01c8dd416270f424ab0c40f9291e269ac6921964)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.