dolthub / dolthub/dolt

Unexpectedly Large Database Disk Usage (24GB) vs Small Backup Size (19MB) - Space Mostly in /.dolt/temptf

Open
#9,983 8 comments 0 reactions 0 assignees View on GitHub
customer issue garbage collection
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 5h
Merged PRs (30d)
108

Description

**Database Name:** insight_metabase
**Version:** 1.75.0
After the application ran for about 10 days, the database file size was approximately 24G. After executing the `dolt gc` command, the database file size was reduced to about 14G.

```
#dolt version
dolt version 1.75.0
```

```
cd /db/dolt/insight_metabase

/db/dolt/insight_metabase#du -h -d 1
24G ./.dolt
24G .
```

After executing the `dolt gc` command:

```
/db/dolt/insight_metabase#dolt gc
/db/dolt/insight_metabase#du -h -d 1
14G ./.dolt
14G .
```

However, our actual data volume is not that large. When I used `dolt backup`, the backup file was very small, only about 19MB.

```
/db/dolt/insight_metabase#mkdir /db/dolt_bak/insight_metabase
/db/dolt/insight_metabase#dolt backup add local-backup file:///db/dolt_bak/insight_metabase
/db/dolt/insight_metabase#dolt backup sync local-backup
/db/dolt/insight_metabase#cd /db/dolt_bak/insight_metabase
/db/dolt_bak/insight_metabase# du -h -d 1
4.0K ./oldgen
19M .
```

Our database has only one branch and one commit.

```
/db/dolt/insight_metabase# dolt branch
* main

/db/dolt/insight_metabase# dolt log
commit g9efkgekkdj74773dclc3t6m35c3m4p1 (HEAD -> main)
Author: Dolt System Account
Date: Wed Oct 22 02:42:03 +0000 2025

Initialize data repository
```

After investigation, we found that the folder `/db/dolt/insight_metabase/.dolt/temptf` occupies almost most of the space.

Our questions are:
1. Why is the disk space usage so large?
2. How can we release the occupied disk space?
3. What can we do to reduce the rate of disk space usage?

The configuration file `config.yaml` we are using is as follows (the `remote_url_template` in `remote_url_template` is partially omitted):

```
data_dir: /db/dolt
cfg_dir: /etc/dolt/config
log_level: info
log_format: text
behavior:
read_only: false
autocommit: true
disable_client_multi_statements: false
dolt_transaction_commit: false
event_scheduler: "ON"
auto_gc_behavior:
enable: true
archive_level: 1
listener:
host: 0.0.0.0
port: 3306
max_connections: 1000
back_log: 50
max_connections_timeout_millis: 60000
read_timeout_millis: 28800000
write_timeout_millis: 28800000
tls_key: null
tls_cert: null
require_secure_transport: null
allow_cleartext_passwords: null

max_logged_query_len: 0

# Advanced Configuration
metrics:
labels: {}
host: null
port: -1

system_variables: {}

user_session_vars: []

jwks: []
cluster:
standby_remotes:
- name: standby
remote_url_template: http://......:50051/{database}
bootstrap_role: primary
bootstrap_epoch: 1
remotesapi:
port: 50051
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.