dgraph-io / dgraph-io/badger

GC doesn't seem to run

Open
#2,003 10 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug Stale
Dominant language
Go
Stars
15.8k
Forks
1.3k
Avg merge
2m
Merged PRs (30d)
1

Description

What version of Badger are you using?

Latest v4

What version of Go are you using?

1.20.3

Have you tried reproducing the issue with the latest release?

Yes

What is the hardware spec (RAM, CPU, OS)?

16gb ram, i5 intel, mac os

What steps will reproduce the bug?
package main

import (
	"fmt"
	"log"
	"math/rand"
	"strconv"
	"time"

	"github.com/dgraph-io/badger/v4"
)

var db *badger.DB

const charset = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"

func generateRandomString(length int) string {
	b := make([]byte, length)
	for i := range b {
		b[i] = charset[rand.Intn(len(charset))]
	}
	return string(b)
}

func main() {
	opts := badger.DefaultOptions("badger")
	opts.ValueLogFileSize = 1 << 20
	var err error
	db, err = badger.Open(opts)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

	fmt.Println("Start inserting 5 million items.")

	for i := 0; i < 5_000_000; i++ {
		err := db.Update(func(txn *badger.Txn) error {
			key := strconv.Itoa(i)
			value := []byte(generateRandomString(300))

			e := badger.NewEntry([]byte(key), value).WithTTL(30 * time.Second)

			err := txn.SetEntry(e)
			if err != nil {
				return err
			}
			return nil
		})

		if err != nil {
			log.Fatal("Error while inserting:", err)
		}
	}

	fmt.Println("Successfully inserted 5 million items.")

	startGarbageCollection()
}

func startGarbageCollection() {
	ticker := time.NewTicker(10 * time.Second)
	defer ticker.Stop()
	for range ticker.C {
	again:
		err := db.RunValueLogGC(0.01)
		fmt.Println(err)
		if err == nil {
			goto again
		}
	}
}
Expected behavior and actual result.

Since the keys are expiring in 30 seconds, I thought that the files in badger/ would get removed.

However, heres what I see:

image image
Additional information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provided Go reproducer and the db.RunValueLogGC entry point, then compare the observed retention of expired entries and value-log files with Badger's documented GC behavior. Done means determining whether the result is expected or defective and, if defective, identifying a focused fix and test for the reported case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.