influxdata / influxdata/influxdb

[v3] InfluxDB3 imported data with plugin - compaction not working

Open
#27,301 26 comments 0 reactions 0 assignees View on GitHub
v3 v3-enterprise
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I am running influxDB3 enterprise Home license on a docker setup. Currently running image 3.8.4

I'm trying to import historic data from an influxDB2 server for a measurement using the "official" import data plugin (https://github.com/influxdata/influxdb3_plugins/blob/main/influxdata/import/README.md).

It imports fine but the historic data doesn't seem to compact properly afterwards and soon causes fan out problems etc.

Currently I've imported one year of data from influxDB2 every 15 seconds or so, and I have 61,000 parquet files, but the number never goes down.

I have tried all manner of config settings, turning off ingest, many different parameters to no avail.

__Steps to reproduce:__
- Add the import data plugin
- Import a year's worth of data 2025-2026 with c. 20 fields at 15second intervals.
- Wait for compaction

__Expected behaviour:__
The number of parquet files should go down over time as compaction runs, evidenced by running this command:
`docker exec influxdb3-enterprise find /var/lib/influxdb3/data -name "*.parquet" | wc -l`

(returns 61916 files)

__Actual behaviour:__
The number of files doesn't go down over time, it stays the same at 61916

__Environment info:__

my current key docker compose params:

command:
- influxdb3
- serve
- --max-http-request-size=150000000
- --node-id=node1
- --cluster-id=cluster0
- --object-store=file
- --data-dir=/var/lib/influxdb3/data
- --plugin-dir=/var/lib/influxdb3/plugins
- --license-email=xxx@xxx
- --license-type=home
- --wal-replay-concurrency-limit=4 # Serial replay: slower but MUCH lower risk of hang/OOM/contention
- --wal-replay-fail-on-error # Crash loudly on bad file instead of silent spin
- --checkpoint-interval=1h # Enable checkpointing NOW (consolidates snapshots monthly-ish; huge future speedup)
- --mode=all
- --compaction-check-interval=5s # Check more often (default 10s)
- --compaction-max-num-files-per-plan=2000 # Larger jobs (default 500; test, watch for OOM)
- --compaction-multipliers=2,4,8,16 # Faster aging/merging (default more conservative)
- --compaction-row-limit=500000
- --force-snapshot-mem-threshold=20MB # Flush/compact earlier buffers
- --datafusion-num-threads=4
- --gen1-lookback-duration=8760h
- --compaction-gen2-duration=1d
environment:
- INFLUXDB3_ENTERPRISE_LICENSE_EMAIL=${INFLUXDB_EMAIL}
- INFLUXDB3_LOG_FILTER=warn,influxdb3_enterprise_compactor=debug
- INFLUXDB3_DATAFUSION_MAX_PARQUET_FANOUT=130000

__Logs:__

$ docker exec influxdb3-enterprise sh -c 'curl -s "http://localhost:8181/api/v3/query_sql?db=farm_data&q=select+*+from+system.compaction_events+where+event_type='"'"'plan_group_run_completed'"'"'+order+by+event_time+desc+limit+10" -H "Authorization: Bearer TOKEN"' | python3 -m json.tool
[
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:58.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.747Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
},
{
"event_time": "2026-03-24T19:13:53.746Z",
"event_type": "plan_group_run_completed",
"event_duration": 0,
"event_status": "success",
"event_data": "{\"plans_ran\":[]}"
}
]

$ docker exec influxdb3-enterprise find /var/lib/influxdb3/data/node1/dbs/1/1 -name "*.parquet" | grep -oP '\d{4}-\d{2}-\d{2}' | sort | uniq -c | sort -rn | head -20
190 2026-03-24
145 2026-02-26
145 2026-02-24
145 2026-02-21
145 2026-02-19
145 2026-02-18
145 2026-02-16
145 2026-02-14
145 2026-02-12
145 2026-02-10
145 2026-02-08
145 2026-02-06
145 2026-02-04
145 2026-02-02
145 2026-01-29
145 2026-01-27
145 2026-01-25
145 2026-01-24
145 2026-01-22
145 2026-01-19

Contributor guide

Open the contributing guide

Research direction

Start with the import plugin README and reproduce the reported Docker setup using the listed compaction settings. Inspect system.compaction_events for plan_group_run_completed entries and compare the parquet-file count command before and after compaction. Done means imported files are compacted and completed events show plans running rather than empty plans.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.