influxdata / influxdata/influxdb
Copying Data from an old autogen retention policy to a new 52w policy does not copy data older than policy creation
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Create an new RP `CREATE RETENTION POLICY "a_year" ON "haproxy" DURATION 52w REPLICATION 1 DEFAULT`
2. copy data from old autogen policy to new one. `SELECT * INTO "haproxy"."a_year"."haproxy" FROM "haproxy"."autogen"."haproxy" GROUP BY *`
3. verify that the data was copied. Data older than creation of new policy does not exist in the new policy.
4. double check by running query in cli: `SELECT mean("scur") FROM "a_year"."haproxy" WHERE ("environment" =~ /^prod$/ AND "proxy" = 'puppet') AND time >= now() - 90d and time < now()-30d GROUP BY time(1h) fill(null)` that query only shows no data for the "a_year" RP. but `SELECT mean("scur") FROM "autogen"."haproxy" WHERE ("environment" =~ /^prod$/ AND "proxy" = 'puppet') AND time >= now() - 90d and time < now()-30d GROUP BY time(1h) fill(null)`, show data.
__Expected behavior:__
copying data from old RP to new RP should populate new RP with data.
__Actual behavior:__
data does not copy to the New RP, and i get a error message saying some data was dropped.
`ERR: partial write: points beyond retention policy dropped=7968`
even though none of the data in the old autogen RP is older than 52w
running the following query (suggest by Jamie Milton in slack): `SELECT * INTO "haproxy"."a_year"."haproxy" FROM "haproxy"."autogen"."haproxy" where time > now()-51w GROUP BY *` does actually copy the data into the new RP. but very slowly.
__Environment info:__
* System info: Linux 4.4.0-186-generic x86_64
* InfluxDB version: InfluxDB v1.8.1 (git: 1.8 af0237819ab9c5997c1c0144862dc762b9d8fc25)
* Other relevant environment details: Linux Ubuntu 16.04.
__Config:__
Copy any non-default config values here or attach the full config as a gist or file.
```
194 [retention]
195 # Determines whether retention policy enforcement enabled.
196 enabled = true
197
198 # The interval of time when retention policy enforcement checks run.
199 check-interval = "30m"
```
__Logs:__
Include snippet of errors in log.
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
```sh
# Commands should be run when the bug is actively happening.
# Note: This command will run for at least 30 seconds.
curl -o profiles.tar.gz "http://localhost:8086/debug/pprof/all?cpu=true"
curl -o vars.txt "http://localhost:8086/debug/vars"
iostat -xd 1 30 > iostat.txt
# Attach the `profiles.tar.gz`, `vars.txt`, and `iostat.txt` output files.
```
Contributor guide
Research direction
Start by reproducing the SELECT INTO from the old autogen retention policy to the new 52w policy, including the reported partial-write error. Compare it with the same query restricted to time > now()-51w and verify both retention policies afterward. Done means eligible data copies without points being incorrectly dropped.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100