influxdata / influxdata/influxdb

Retention policy cleanup remove series from index memery failed

Open
#10,476 5 comments 1 reaction 0 assignees View on GitHub
1.x area/storage kind/bug
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__System info:InfluxDB V1.6.4 CentOS Linux release 7.5.1804

__Steps to reproduce:__

1. Create database DB0, set RP to duration 2h shard duration 1h
2. Insert datas that series is different to DB0, and 'show series cardinary' return 10
3. after 2h , after do RP, 'show series cardinary' result is 0,
4. dlv attach 'the influxdb process ID'
b github.com/influxdata/influxdb/tsdb/index/inmem.(*Index).CreateSeriesListIfNotExists
insert any point that can create new series then go to break
p len(i.series)
the result is 9

__Expected behavior: index memery len(series) is same as 'show series cardinary'

__Actual behavior: index memery len(series) is big than 'show series cardinary', it will cause series is not exceed the limit but can not add new.

It may be the reason:
tsdb/store.go
```
// DeleteShard removes a shard from disk.
func (s *Store) DeleteShard(shardID uint64) error {
sh := s.Shard(shardID)
......
if ss.Cardinality() > 0 {
sfile := s.seriesFile(db)
if sfile != nil {
// If the inmem index is in use, then the series being removed from the
// series file will also need to be removed from the index.
if index.Type() == InmemIndexName {
var keyBuf []byte // Series key buffer.
var name []byte
var tagsBuf models.Tags // Buffer for tags container.
var err error

ss.ForEach(func(id uint64) {
skey := sfile.SeriesKey(id) // Series File series key
if skey == nil {
return
}

name, tagsBuf = ParseSeriesKeyInto(skey, tagsBuf)
/*********************************************************
keyBuf will always append tagsBuf, so it is wrong value, in DropSeriesGlobal can not drop.
********************************************************/
keyBuf = models.AppendMakeKey(keyBuf, name, tagsBuf)
if err = index.DropSeriesGlobal(keyBuf); err != nil {
return
}
})

if err != nil {
return err
}
}

ss.ForEach(func(id uint64) {
sfile.DeleteSeriesID(id)
})
}
```

**Please note** It will take at least 30 seconds for the first cURL command above to return a response.
This is because it will run a CPU profile as part of its information gathering, which takes 30 seconds to collect.
Ideally you should run these commands when you're experiencing problems, so we can capture the state of the system at that time.

If you're concerned about running a CPU profile (which only has a small, temporary impact on performance), then you can set `?cpu=false` or omit `?cpu=true` altogether.

Please run those if possible and link them from a [gist](http://gist.github.com) or simply attach them as a comment to the issue.

*Please note, the quickest way to fix a bug is to open a Pull Request.*

Contributor guide

Open the contributing guide

Research direction

Start in tsdb/store.go at Store.DeleteShard and follow the in-memory index path through Index.DropSeriesGlobal and CreateSeriesListIfNotExists. Reproduce the retention-policy sequence, inspect the series key handling, and compare the in-memory series count with SHOW SERIES CARDINALITY after shard deletion. Done means expired series are removed from the index so the counts agree and new series can be added.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.