Improve memory configuration documentation and validation for InfluxDB3 Enterprise

---
title: "Improve memory configuration documentation and validation for InfluxDB3 Enterprise"
labels:
- documentation
- enhancement
- influxdb3-enterprise
---

## Summary

Add comprehensive memory configuration documentation and validation to prevent common OOM (Out of Memory) kills in InfluxDB 3 Enterprise deployments, particularly affecting the compactor component.

## Problem

Customer deployments are experiencing OOM kills due to memory overcommitment from misconfigured cache settings. Current issues:

1. **No memory configuration guidance** in HELM chart documentation
2. **No validation** of memory settings during chart installation
3. **Percentage-based cache configuration is ambiguous** (`INFLUXDB3_PT_FILE_CACHE_MAX_BYTES: 50%`)
4. **Conflicting defaults** that lead to cache allocation > container limit
5. **No warnings** when force snapshot threshold is misconfigured

### Real-World Example (VAST Customer)

Customer configured:
```yaml
resources:
limits:
memory: 95Gi

env:
- name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
value: 50% # 47.5GB
- name: INFLUXDB3_OBJECT_STORE_CACHE_MAX_SIZE_BYTES
value: "34359738368" # 32GB
- name: INFLUXDB3_PT_COMPACTOR_INPUT_SIZE_BUDGET
value: 12GB
```

**Result:**
- Total cache allocation: 91.5GB (96% of limit)
- Available for operations: 3.5GB (4% of limit)
- **Compactor OOMKilled after 38 minutes**

**Root cause:** Caches alone consumed 96% of available memory, leaving insufficient space for compaction operations.

## Proposed Solution

### 1. Add Memory Sizing Documentation

Create comprehensive memory sizing guide covering:
- Memory budget calculation
- Component-by-component sizing guidelines
- Pre-configured profiles (small/standard/large/high-performance)
- Troubleshooting guide
- Validation checklist

**Documentation structure:**
```
docs/
memory-sizing-guide.md # Comprehensive guide with calculator
memory-troubleshooting.md # OOM troubleshooting
examples/
memory-config-64gb.yaml # Small deployment
memory-config-95gb.yaml # Standard deployment
memory-config-150gb.yaml # Large deployment
memory-config-200gb.yaml # High-performance deployment
```

### 2. Add HELM Chart Validation

Add `values.schema.json` to validate memory configuration at install/upgrade time.

**Validation checks:**
1. Warn if total cache allocation > 80% of container memory
2. Error if force snapshot threshold > 60% of container memory
3. Warn if force snapshot threshold > total cache allocation
4. Warn if percentage-based cache configuration used in production
5. Recommend absolute byte values instead of percentages

**Example validation output:**
```
⚠️ WARNING: Memory overcommitment detected
Total cache allocation: 91.5GB (96% of 95GB limit)
Recommended maximum: 76GB (80% of limit)

Current configuration:
File cache: 47.5GB (50%)
Object store cache: 32.0GB
Compactor input budget: 12.0GB

Recommendation:
Use one of the pre-configured memory profiles:
helm install ... -f examples/memory-config-95gb.yaml

Or reduce cache sizes manually. See:
docs/memory-sizing-guide.md
```

### 3. Update Default Values

Change `values.yaml` defaults to be safer:

**Current (dangerous):**
```yaml
compactor:
env:
- name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
value: 50% # Ambiguous, error-prone
- name: INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD
value: 70% # Too high
```

**Proposed (safe):**
```yaml
compactor:
env:
# Use absolute values by default (for 95GB container)
- name: INFLUXDB3_PT_FILE_CACHE_MAX_BYTES
value: "21474836480" # 20GB (21% of 95GB)

- name: INFLUXDB3_OBJECT_STORE_CACHE_MAX_SIZE_BYTES
value: "21474836480" # 20GB (21% of 95GB)

- name: INFLUXDB3_PT_COMPACTOR_INPUT_SIZE_BUDGET
value: "8GB"

- name: INFLUXDB3_EXEC_MEM_POOL_BYTES
value: "21474836480" # 20GB (21% of 95GB)

- name: INFLUXDB3_DATAFUSION_NUM_THREADS
value: "8" # Limit parallelism

- name: INFLUXDB3_FORCE_SNAPSHOT_MEM_THRESHOLD
value: "50%" # Reduced from 70%
```

**Memory budget with safe defaults:**
```
File cache: 20 GB (21%)
Object store cache: 20 GB (21%)
Compactor input budget: 8 GB (8%)
Exec mem pool: 20 GB (21%)
Runtime overhead: 5 GB (5%)
---------
Total allocated: 73 GB (77%)
Container limit: 95 GB
Available for ops: 22 GB (23%) ✅ SAFE
```

### 4. Add Pre-Configured Memory Profiles

Add example configurations for common deployment sizes:

**`examples/memory-config-64gb.yaml`** (Small deployment)
- Container: 64GB
- File cache: 15GB
- Object cache: 10GB
- Input budget: 5GB
- Exec pool: 10GB
- Total allocated: 43GB (67%)

**`examples/memory-config-95gb.yaml`** (Standard deployment)
- Container: 95GB
- File cache: 20GB
- Object cache: 20GB
- Input budget: 8GB
- Exec pool: 20GB
- Total allocated: 73GB (77%)

**`examples/memory-config-150gb.yaml`** (Large deployment)
- Container: 150GB
- File cache: 40GB
- Object cache: 30GB
- Input budget: 15GB
- Exec pool: 30GB
- Total allocated: 123GB (82%)

**`examples/memory-config-200gb.yaml`** (High-performance deployment)
- Container: 200GB
- File cache: 50GB
- Object cache: 40GB
- Input budget: 20GB
- Exec pool: 40GB
- Total allocated: 160GB (80%)

Usage:
```bash
helm install influxdb3-enterprise influxdata/influxdb3-enterprise \
-f examples/memory-config-95gb.yaml \
-f my-custom-values.yaml
```

### 5. Add Memory Monitoring Dashboard

Add Grafana dashboard with memory metrics:
- Container memory usage (%)
- Cache utilization (file, object)
- Execution pool utilization
- Forced snapshot rate
- OOMKill event count

**Alert thresholds:**
- ⚠️ Warning at 80% memory usage
- 🔥 Critical at 90% memory usage
- 🔥 Critical on any OOMKill event

## Implementation Checklist

### Phase 1: Documentation (High Priority)
- [ ] Create `docs/memory-sizing-guide.md`
- [ ] Create `docs/memory-troubleshooting.md`
- [ ] Update main README with memory sizing section
- [ ] Add memory configuration to troubleshooting guide

### Phase 2: Example Configurations (High Priority)
- [ ] Create `examples/memory-config-64gb.yaml`
- [ ] Create `examples/memory-config-95gb.yaml`
- [ ] Create `examples/memory-config-150gb.yaml`
- [ ] Create `examples/memory-config-200gb.yaml`

### Phase 3: Chart Validation (Medium Priority)
- [ ] Add `values.schema.json` with memory validation rules
- [ ] Implement validation logic in `_helpers.tpl`
- [ ] Add memory budget calculator helper function
- [ ] Display warnings during `helm install`/`upgrade`

### Phase 4: Update Defaults (Medium Priority)
- [ ] Change default cache values to absolute bytes
- [ ] Reduce force snapshot threshold from 70% to 50%
- [ ] Add `INFLUXDB3_EXEC_MEM_POOL_BYTES` to defaults
- [ ] Add `INFLUXDB3_DATAFUSION_NUM_THREADS` to defaults

### Phase 5: Monitoring (Low Priority)
- [ ] Create Grafana dashboard JSON
- [ ] Add Prometheus recording rules for memory metrics
- [ ] Add alerting rules for OOMKill events
- [ ] Document dashboard installation

## Testing Plan

1. **Validation testing:**
- Test that validation catches overcommitted memory
- Test that validation warns about percentage-based configs
- Test that validation accepts safe configurations

2. **Configuration testing:**
- Deploy with each pre-configured profile
- Run compaction workload for 2+ hours
- Verify no OOM kills
- Verify memory stays under 85% of limit

3. **Upgrade testing:**
- Upgrade existing deployment with new defaults
- Verify no disruption to running workload
- Verify memory usage improves

## Benefits

1. **Prevents OOM kills** by providing safe default configurations
2. **Educates users** with comprehensive documentation
3. **Validates early** to catch misconfigurations before deployment
4. **Saves time** with pre-configured profiles for common scenarios
5. **Improves observability** with memory monitoring dashboard

## Related Issues

- #800 - Initial VAST customer issue (startup probe timeout)
- #801 - PR fixing startup probe timeout
- Comment: https://github.com/influxdata/helm-charts/pull/801#issuecomment-4791657797

## Additional Context

Full analysis and documentation already created:
- `COMPACTOR-OOM-ANALYSIS.md` - Deep technical analysis (23KB)
- `COMPACTOR-OOM-QUICK-FIX.md` - Customer quick fix guide (19KB)
- `MEMORY-SIZING-GUIDE.md` - Comprehensive sizing guide (18KB)

These documents are available and can be adapted for inclusion in the HELM chart repository.

Review the existing Helm chart, values.yaml, README, and the named COMPACTOR-OOM-ANALYSIS.md, COMPACTOR-OOM-QUICK-FIX.md, and MEMORY-SIZING-GUIDE.md documents first. This issue bundles documentation, example YAML profiles, chart validation, default changes, monitoring, and testing; completion requires implementing and verifying the selected phases from the checklist.

Written by the indexing model from the issue text.