Automatic garbage collection not running during long `LOAD DATA INFILE`
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 120
Description
```
$ ls -alh .dolt/noms
total 294349328
drwxr-xr-x 8 timsehn staff 256B Feb 21 11:24 .
drwxr-xr-x 8 timsehn staff 256B Feb 21 13:06 ..
-rw-r--r-- 1 timsehn staff 1.5G Feb 21 13:09 journal.idx
-rw------- 1 timsehn staff 0B Feb 20 09:20 LOCK
-rw------- 1 timsehn staff 151B Feb 21 11:23 manifest
-rw------- 1 timsehn staff 41G Feb 21 11:23 o03nl0fkb3cagtb84vhlnua9rudnofdb.darc
drwxr-xr-x 5 timsehn staff 160B Feb 20 10:07 oldgen
-rw-r--r-- 1 timsehn staff 98G Feb 21 13:09 vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
$ dolt sql -q "show processlist"
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| 70 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 67 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 68 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 81 | root | localhost | doge_medicaid | Query | 8014 | | LOAD DATA LOCAL INFILE '/tmp/medicaid-provider-spending.csv' INTO TABLE medicaid_provider_spending FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 LINES (BILLING_PROVIDER_NPI_NUM, SERVICING_PROVIDER_NPI_NUM, HCPCS_CODE, CLAIM_FROM_MONTH, TOTAL_UNIQUE_BENEFICIARIES, TOTAL_CLAIMS, TOTAL_PAID) |
| | | | | | | medicaid_provider_spending (0/? partitions) | |
| | | | | | | | |
| 96 | __dolt_local_user__ | localhost | doge_medicaid | Query | 0 | running | show processlist |
| 66 | root | localhost | doge_medicaid | Sleep | 5693 | | |
| 69 | root | localhost | doge_medicaid | Sleep | 5693 | | |
+----+---------------------+-----------+---------------+---------+------+---------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
```
I have a 98GB journal file.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the long LOAD DATA LOCAL INFILE operation shown in the issue and monitor the .dolt/noms journal and process list while it runs. Investigate how garbage collection behaves during this operation; done means automatic garbage collection runs without leaving a journal file of the reported size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100