influxdata / influxdata/docs-v2
docs(influxdb3): document write path stages, storage growth, and fix "snapshot files" terminology
- Dominant language
- JavaScript
- Stars
- 82
- Forks
- 326
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 82
Description
### Problem
The InfluxDB 3 Core/Enterprise documentation lacks coverage of:
1. **Write path stages** — How data flows from WAL → snapshot → gen1 parquet files. The existing [durability page](/influxdb3/core/reference/internals/durability/) describes the high-level flow but doesn't explain what snapshot files or gen1 parquet files actually are, where they live in object storage, or how they relate to each other.
2. **Storage growth patterns** — Core accumulates gen1 parquet files indefinitely with no automatic cleanup. Enterprise uses a compactor and `Gen1CleanupHandler`. Users have no documentation to understand why their object storage is growing or what (if anything) they can do about it.
3. **Terminology inconsistency** — The backup-restore page uses "Table snapshot files" as a heading and table label, which can be confused with "snapshot files" (the recovery-point metadata in `/snapshots/`). These are different file types in different paths.
### Proposed changes
#### 1. Rename "Table snapshot files" → "Table snapshots"
**File:** `content/shared/influxdb3-admin/backup-restore.md`
- Rename the `### Table snapshot files` heading (line 496) to `### Table snapshots`
- Update the two table rows (lines 33 and 50) from "Table snapshot files" to "Table snapshots"
- Fix any links targeting the `#table-snapshot-files` anchor (currently none exist, but verify before merging)
This aligns with the convention established in PR #6896, where "snapshots" and "checkpoints" are used as shorthand without "files."
#### 2. Expand the durability page with write path details
**File:** `content/shared/influxdb3-internals-reference/durability.md`
Add sections covering:
- **Snapshot process**: What happens when `wal-snapshot-size` WAL files accumulate — the server persists the in-memory buffer as gen1 parquet files and writes a snapshot metadata file (`.info.json`) recording which parquet files were created
- **Object store layout**: Directory structure (`wal/`, `snapshots/`, `dbs/`, `snapshot-checkpoints/`, `table-snapshots/`, `db-indices/`)
- **File naming**: Inverted sequence numbers for newest-first listing
#### 3. Document storage growth and cleanup
**File:** `content/shared/influxdb3-admin/performance-tuning.md` (new section) or a dedicated page
Add sections covering:
| File type | Growth | Automatic cleanup |
|:----------|:-------|:------------------|
| WAL files | Bounded by `--snapshotted-wal-files-to-keep` | Yes |
| Snapshot metadata | Unbounded (one per snapshot event) | No |
| Checkpoints | Bounded (~2 per month) | No, but growth is slow |
| Gen1 parquet (Core) | Unbounded | No |
| Gen1 parquet (Enterprise) | Bounded after compaction | Yes (compactor + cleanup) |
Key points to document:
- `--gen1-lookback-duration` controls what's loaded into memory, not what's stored
- `--checkpoint-interval` doesn't delete old snapshots
- No `influxdb3` CLI command exists for manual pruning
- Core vs Enterprise differences in gen1 file lifecycle
#### 4. Add troubleshooting guidance
Add to the performance-tuning or a troubleshooting page:
- **Slow startup**: Check snapshot count, enable `--checkpoint-interval`, tune `--gen1-lookback-duration`
- **Growing object storage (Core)**: Gen1 parquet accumulation, no automated pruning
- **WAL files not cleaning up**: Check `--snapshotted-wal-files-to-keep`, verify snapshots are happening
### Context
- PR #6896 documented `--checkpoint-interval` and established the "snapshots"/"checkpoints" shorthand convention
- The write path details come from source code analysis of InfluxDB 3 Core v3.8.2
- Some details (default values, Enterprise compactor behavior) should be verified against current source before documenting
### Tasks
- [ ] Rename "Table snapshot files" → "Table snapshots" in `content/shared/influxdb3-admin/backup-restore.md` (heading + two table rows)
- [ ] Verify no links target `#table-snapshot-files` anchor
- [ ] Expand durability page with snapshot process and object store layout
- [ ] Document storage growth patterns (Core vs Enterprise)
- [ ] Add troubleshooting section for startup, storage growth, and WAL cleanup
- [ ] Document how to manually clean up gen1 parquet files and old snapshots from object storage (Core has no automatic cleanup — confirm safe procedures with engineering)
```
There is an undocumented admin endpoint for it, which an LLM audit caught yesterday: https://github.com/influxdata/docs-v2/issues/6900 (see near the end of the LLM response summary, the section "Internal-Only Endpoints").
_Originally posted by @waynr in https://github.com/influxdata/docs-v2/pull/6896#discussion_r2886876911_
```
- [ ] Verify default values and Enterprise compactor behavior against current source
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with content/shared/influxdb3-admin/backup-restore.md, content/shared/influxdb3-internals-reference/durability.md, and content/shared/influxdb3-admin/performance-tuning.md. Review PR #6896, issue #6900, and current source to verify terminology, defaults, Enterprise compactor behavior, and safe cleanup guidance. Done means the requested naming, write-path and object-store explanations, growth guidance, troubleshooting, and cleanup details are documented and the checklist is complete.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, documentation
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100