influxdata / influxdata/influxdb
[v3] InfluxDB3 imported data with plugin - compaction not working
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
I am running influxDB3 enterprise Home license on a docker setup. Currently running image 3.8.4
I'm trying to import historic data from an influxDB2 server for a measurement using the "official" import data plugin (https://github.com/influxdata/influxdb3_plugins/blob/main/influxdata/import/README.md).
It imports fine but the historic data doesn't seem to compact properly afterwards and soon causes fan out problems etc.
Currently I've imported one year of data from influxDB2 every 15 seconds or so, and I have 61,000 parquet files, but the number never goes down.
I have tried all manner of config settings, turning off ingest, many different parameters to no avail.
__Steps to reproduce:__
- Add the import data plugin
- Import a year's worth of data 2025-2026 with c. 20 fields at 15second intervals.
- Wait for compaction
__Expected behaviour:__
The number of parquet files should go down over time as compaction runs, evidenced by running this command:
`docker exec influxdb3-enterprise find /var/lib/influxdb3/data -name "*.parquet" | wc -l`
(returns 61916 files)
__Actual behaviour:__
The number of files doesn't go down over time, it stays the same at 61916
__Environment info:__
my current key docker compose params:
command:
- influxdb3
- serve
- --max-http-request-size=150000000
- --node-id=node1
- --cluster-id=cluster0
- --object-store=file
- --data-dir=/var/lib/influxdb3/data
- --plugin-dir=/var/lib/influxdb3/plugins
- --license-email=xxx@xxx
- --license-type=home
- --wal-replay-concurrency-limit=4 # Serial replay: slower but MUCH lower risk of hang/OOM/contention
- --wal-replay-fail-on-error # Crash loudly on bad file instead of silent spin
- --checkpoint-interval=1h # Enable checkpointing NOW (consolidates snapshots monthly-ish; huge future speedup)
- --mode=all
- --compaction-check-interval=5s # Check more often (default 10s)
- --compaction-max-num-files-per-plan=2000 # Larger jobs (default 500; test, watch for OOM)
- --compaction-multipliers=2,4,8,16 # Faster aging/merging (default more conservative)
- --compaction-row-limit=500000
- --force-snapshot-mem-threshold=20MB # Flush/compact earlier buffers
- --datafusion-num-threads=4
- --gen1-lookback-duration=8760h
- --compaction-gen2-duration=1d
environment:
- INFLUXDB3_ENTERPRISE_LICENSE_EMAIL=${INFLUXDB_EMAIL}
- INFLUXDB3_LOG_FILTER=warn,influxdb3_enterprise_compactor=debug
- INFLUXDB3_DATAFUSION_MAX_PARQUET_FANOUT=130000
__Logs:__
$ docker exec influxdb3-enterprise sh -c 'curl -s "http://localhost:8181/api/v3/query_sql?db=farm_data&q=select+*+from+system.compaction_events+where+event_type='"'"'plan_group_run_completed'"'"'+order+by+event_time+desc+limit+10" -H "Authorization: Bearer TOKEN"' | python3 -m json.tool
[
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
}
]
$ docker exec influxdb3-enterprise find /var/lib/influxdb3/data/node1/dbs/1/1 -name "*.parquet" | grep -oP '\d{4}-\d{2}-\d{2}' | sort | uniq -c | sort -rn | head -20
190 2026-03-24
145 2026-02-26
145 2026-02-24
145 2026-02-21
145 2026-02-19
145 2026-02-18
145 2026-02-16
145 2026-02-14
145 2026-02-12
145 2026-02-10
145 2026-02-08
145 2026-02-06
145 2026-02-04
145 2026-02-02
145 2026-01-29
145 2026-01-27
145 2026-01-25
145 2026-01-24
145 2026-01-22
145 2026-01-19
Contributor guide
Research direction
Start with the import plugin README and reproduce the reported Docker setup using the listed compaction settings. Inspect system.compaction_events for plan_group_run_completed entries and compare the parquet-file count command before and after compaction. Done means imported files are compacted and completed events show plans running rather than empty plans.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100