influxdata / influxdata/docs-v2

docs(influxdb3): document write path stages, storage growth, and fix "snapshot files" terminology

Open
#6,903 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
JavaScript
Stars
82
Forks
326
Avg merge
1d 1h
Merged PRs (30d)
82

Description

### Problem

The InfluxDB 3 Core/Enterprise documentation lacks coverage of:

1. **Write path stages** — How data flows from WAL → snapshot → gen1 parquet files. The existing [durability page](/influxdb3/core/reference/internals/durability/) describes the high-level flow but doesn't explain what snapshot files or gen1 parquet files actually are, where they live in object storage, or how they relate to each other.

2. **Storage growth patterns** — Core accumulates gen1 parquet files indefinitely with no automatic cleanup. Enterprise uses a compactor and `Gen1CleanupHandler`. Users have no documentation to understand why their object storage is growing or what (if anything) they can do about it.

3. **Terminology inconsistency** — The backup-restore page uses "Table snapshot files" as a heading and table label, which can be confused with "snapshot files" (the recovery-point metadata in `/snapshots/`). These are different file types in different paths.

### Proposed changes

#### 1. Rename "Table snapshot files" → "Table snapshots"

**File:** `content/shared/influxdb3-admin/backup-restore.md`

- Rename the `### Table snapshot files` heading (line 496) to `### Table snapshots`
- Update the two table rows (lines 33 and 50) from "Table snapshot files" to "Table snapshots"
- Fix any links targeting the `#table-snapshot-files` anchor (currently none exist, but verify before merging)

This aligns with the convention established in PR #6896, where "snapshots" and "checkpoints" are used as shorthand without "files."

#### 2. Expand the durability page with write path details

**File:** `content/shared/influxdb3-internals-reference/durability.md`

Add sections covering:

- **Snapshot process**: What happens when `wal-snapshot-size` WAL files accumulate — the server persists the in-memory buffer as gen1 parquet files and writes a snapshot metadata file (`.info.json`) recording which parquet files were created
- **Object store layout**: Directory structure (`wal/`, `snapshots/`, `dbs/`, `snapshot-checkpoints/`, `table-snapshots/`, `db-indices/`)
- **File naming**: Inverted sequence numbers for newest-first listing

#### 3. Document storage growth and cleanup

**File:** `content/shared/influxdb3-admin/performance-tuning.md` (new section) or a dedicated page

Add sections covering:

| File type | Growth | Automatic cleanup |
|:----------|:-------|:------------------|
| WAL files | Bounded by `--snapshotted-wal-files-to-keep` | Yes |
| Snapshot metadata | Unbounded (one per snapshot event) | No |
| Checkpoints | Bounded (~2 per month) | No, but growth is slow |
| Gen1 parquet (Core) | Unbounded | No |
| Gen1 parquet (Enterprise) | Bounded after compaction | Yes (compactor + cleanup) |

Key points to document:
- `--gen1-lookback-duration` controls what's loaded into memory, not what's stored
- `--checkpoint-interval` doesn't delete old snapshots
- No `influxdb3` CLI command exists for manual pruning
- Core vs Enterprise differences in gen1 file lifecycle

#### 4. Add troubleshooting guidance

Add to the performance-tuning or a troubleshooting page:

- **Slow startup**: Check snapshot count, enable `--checkpoint-interval`, tune `--gen1-lookback-duration`
- **Growing object storage (Core)**: Gen1 parquet accumulation, no automated pruning
- **WAL files not cleaning up**: Check `--snapshotted-wal-files-to-keep`, verify snapshots are happening

### Context

- PR #6896 documented `--checkpoint-interval` and established the "snapshots"/"checkpoints" shorthand convention
- The write path details come from source code analysis of InfluxDB 3 Core v3.8.2
- Some details (default values, Enterprise compactor behavior) should be verified against current source before documenting

### Tasks

- [ ] Rename "Table snapshot files" → "Table snapshots" in `content/shared/influxdb3-admin/backup-restore.md` (heading + two table rows)
- [ ] Verify no links target `#table-snapshot-files` anchor
- [ ] Expand durability page with snapshot process and object store layout
- [ ] Document storage growth patterns (Core vs Enterprise)
- [ ] Add troubleshooting section for startup, storage growth, and WAL cleanup
- [ ] Document how to manually clean up gen1 parquet files and old snapshots from object storage (Core has no automatic cleanup — confirm safe procedures with engineering)
```
There is an undocumented admin endpoint for it, which an LLM audit caught yesterday: https://github.com/influxdata/docs-v2/issues/6900 (see near the end of the LLM response summary, the section "Internal-Only Endpoints").

_Originally posted by @waynr in https://github.com/influxdata/docs-v2/pull/6896#discussion_r2886876911_
```
- [ ] Verify default values and Enterprise compactor behavior against current source

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with content/shared/influxdb3-admin/backup-restore.md, content/shared/influxdb3-internals-reference/durability.md, and content/shared/influxdb3-admin/performance-tuning.md. Review PR #6896, issue #6900, and current source to verify terminology, defaults, Enterprise compactor behavior, and safe cleanup guidance. Done means the requested naming, write-path and object-store explanations, growth guidance, troubleshooting, and cleanup details are documented and the checklist is complete.

Written by the indexing model from the issue text.

Assessment

Domain
databases, documentation
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.