influxdata / influxdata/influxdb
How to vacuum deleted data, which is still present per `influxd inspect export-lp`?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
Dear InfluxDB team,
first things first: Thanks a stack for conceiving and maintaining InfluxDB.
We are running InfluxDB 2.7.3, and have been tripped by the `influxd inspect export-lp` command. We already searched the issue knowledgebase like https://github.com/influxdata/influxdb/issues?q=is%3Aissue+export-lp, but haven't been able to discover a similar report like this one.
So, it is about the venerable topic of deleting data. We discovered that deleted data appears to be gone when querying it, but it is apparently still present when exporting it using `influxd inspect export-lp`.
On this matter, we are wondering if retention procedures like outlined at [^1] might need to be involved to make the data completely gone, or if there is some other vacuum-like command we may need to invoke to get better consistency. In this case, we are not using any data retention features, just plain delete operations.
It is likely related to this log item when running `influxd inspect export-lp`?
```json
{"level":"warn","ts":1699727385.948989,"caller":"export_lp/export_lp.go:307","msg":"detected deletes in WAL file, some deleted data may be brought back by replaying this export"}
```
Below, we attached a reproduction case, which demonstrates the scenario, and we will be happy about any help on this topic, how to get a consistent dump file in lineprotocol format.
With kind regards,
Andreas.
[^1]: https://docs.influxdata.com/influxdb/v2/reference/internals/data-retention/#when-does-data-actually-get-deleted
### Run InfluxDB
```shell
mkdir -p ./var/lib/influxdb2
docker run --rm -it --publish=8086:8086 \
--env=DOCKER_INFLUXDB_INIT_MODE=setup \
--env=DOCKER_INFLUXDB_INIT_USERNAME=admin \
--env=DOCKER_INFLUXDB_INIT_PASSWORD=secret1234 \
--env=DOCKER_INFLUXDB_INIT_ORG=example \
--env=DOCKER_INFLUXDB_INIT_BUCKET=default \
--env=DOCKER_INFLUXDB_INIT_ADMIN_TOKEN=token \
--volume="$PWD/var/lib/influxdb2:/var/lib/influxdb2" \
influxdb:2.7
```
### Procedure
```shell
export INFLUX_ORG=example
export INFLUX_TOKEN=token
export INFLUX_BUCKET_NAME=foobar
export INFLUX_MEASUREMENT=myMeasurement
# Import data and query for it.
influx bucket create
influx write --precision=s "${INFLUX_MEASUREMENT},host=myHost testField=\"testData\" 1556896326"
influx query "from(bucket:\"${INFLUX_BUCKET_NAME}\") |> range(start:-100y)"
# Delete data, and verify it's gone by querying it.
influx delete --start="1877-09-21T00:12:43.145224194Z" --stop="2162-04-11T23:47:16.854775806Z"
influx query "from(bucket:\"${INFLUX_BUCKET_NAME}\") |> range(start:-100y)"
# Export data in lineprotocol format (ILP), demonstrating it is not actually gone, but just soft-deleted?
BUCKET_ID=$(influx bucket list --name ${INFLUX_BUCKET_NAME} --json | jq -r '.[].id')
influxd inspect export-lp \
--engine-path ./var/lib/influxdb2/engine \
--bucket-id ${BUCKET_ID} \
--measurement ${INFLUX_MEASUREMENT} \
--output-path -
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the supplied Docker setup and command sequence, then inspect the export-lp entry point in export_lp/export_lp.go, especially the warning at line 307. Compare query results with the generated line protocol after deletion. Done means the issue's behavior is explained and export-lp no longer produces an inconsistent dump, or the required cleanup procedure is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, shell
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100