dolthub / dolthub/dolt

Automatic garbage collection not running during long `LOAD DATA INFILE`

Open
#10,545 2 comments 0 reactions 0 assignees View on GitHub
enhancement garbage collection
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 8h
Merged PRs (30d)
120

Description

```
$ ls -alh .dolt/noms
total 294349328
drwxr-xr-x 8 timsehn staff 256B Feb 21 11:24 .
drwxr-xr-x 8 timsehn staff 256B Feb 21 13:06 ..
-rw-r--r-- 1 timsehn staff 1.5G Feb 21 13:09 journal.idx
-rw------- 1 timsehn staff 0B Feb 20 09:20 LOCK
-rw------- 1 timsehn staff 151B Feb 21 11:23 manifest
-rw------- 1 timsehn staff 41G Feb 21 11:23 o03nl0fkb3cagtb84vhlnua9rudnofdb.darc
drwxr-xr-x 5 timsehn staff 160B Feb 20 10:07 oldgen
-rw-r--r-- 1 timsehn staff 98G Feb 21 13:09 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$ dolt sql -q "show processlist"
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 70 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 67 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 68 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 81 | root | localhost | doge_medicaid | Query | 8014 | | LOAD DATA LOCAL INFILE '/tmp/medicaid-provider-spending.csv' INTO TABLE medicaid_provider_spending FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES (BILLING_PROVIDER_NPI_NUM, SERVICING_PROVIDER_NPI_NUM, HCPCS_CODE, CLAIM_FROM_MONTH, TOTAL_UNIQUE_BENEFICIARIES, TOTAL_CLAIMS, TOTAL_PAID) |
| | | | | | | medicaid_provider_spending (0/? partitions) | |
| | | | | | | | |
| 96 | __dolt_local_user__ | localhost | doge_medicaid | Query | 0 | running | show processlist |
| 66 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 69 | root | localhost | doge_medicaid | Sleep | 5693 | | |
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```

I have a 98GB journal file.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the long LOAD DATA LOCAL INFILE operation shown in the issue and monitor the .dolt/noms journal and process list while it runs. Investigate how garbage collection behaves during this operation; done means automatic garbage collection runs without leaving a journal file of the reported size.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.