[Workload]: backup-churn

Workload Name

backup-churn

Workload Description

Continuous file lifecycle workload that creates, modifies, and deletes files on a data disk at configurable rates. Produces sustained filesystem churn — new files appearing, existing files being modified (appended, truncated, rewritten), and old files being deleted — to simulate real-world data mutation patterns that stress backup and disaster recovery products.

This fills a gap distinct from the existing disk workload. The disk workload uses fio for block-level I/O patterns (random reads, sequential writes, mixed I/O) — it stresses the storage driver and measures IOPS/throughput. backup-churn operates at the filesystem level — creating, modifying, and removing actual files with realistic directory structures. Backup/DR products don't back up raw blocks; they back up files and filesystems. The challenge for these products is tracking what changed between backup windows: new files, modified files, deleted files, renamed files. That's exactly what this workload produces.

Tooling and Packages

• Tool: Custom shell script (zero external dependencies)
• RPM packages: none — uses only coreutils (dd, mkdir, rm, mv, truncate, find)
• systemd service command: /usr/local/bin/virtwork-backup-churn.sh
• Configurable parameters:
  • churn-rate: operations per second (default: 5)
  • file-size-min / file-size-max: file size range (default: 1KB–10MB)
  • max-files: maximum files on disk before deletions increase (default: 10000)
  • dir-depth: maximum directory nesting depth (default: 5)
  • data-path: target directory on the data disk (default: /data/churn)

VM Count Model

Single VM (like cpu, memory, disk)

Required Resources

• Persistent storage (DataVolume)
• Kubernetes Service (for inter-VM communication)
• Kubernetes Secret (for credentials or config)
• Additional CPU/memory beyond defaults
• GPU or special device passthrough

Cloud-Init Details

write_files:
  - path: /usr/local/bin/virtwork-backup-churn.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      set -euo pipefail
      DATA_DIR="${CHURN_DATA_PATH:-/data/churn}"
      RATE="${CHURN_RATE:-5}"
      MIN_SIZE="${CHURN_FILE_SIZE_MIN:-1024}"
      MAX_SIZE="${CHURN_FILE_SIZE_MAX:-10485760}"
      MAX_FILES="${CHURN_MAX_FILES:-10000}"
      MAX_DEPTH="${CHURN_DIR_DEPTH:-5}"
      DELAY=$(echo "scale=3; 1/$RATE" | bc)

      mkdir -p "$DATA_DIR"
      FILE_COUNT=0

      random_size() {
        echo $(( (RANDOM % (MAX_SIZE - MIN_SIZE + 1)) + MIN_SIZE ))
      }

      random_subdir() {
        local depth=$(( RANDOM % MAX_DEPTH + 1 ))
        local path="$DATA_DIR"
        for ((i=0; i<depth; i++)); do
          path="$path/dir_$(( RANDOM % 20 ))"
        done
        mkdir -p "$path"
        echo "$path"
      }

      while true; do
        FILE_COUNT=$(find "$DATA_DIR" -type f 2>/dev/null | wc -l)
        OP_RAND=$((RANDOM % 100))

        if [ "$FILE_COUNT" -lt 10 ] || [ "$OP_RAND" -lt 40 ]; then
          # CREATE: 40% of ops (or forced when few files exist)
          DIR=$(random_subdir)
          SIZE=$(random_size)
          dd if=/dev/urandom of="$DIR/file_$(date +%s%N)_$$" bs=1 count="$SIZE" 2>/dev/null
        elif [ "$OP_RAND" -lt 70 ]; then
          # MODIFY: 30% of ops — append, truncate, or rewrite
          TARGET=$(find "$DATA_DIR" -type f 2>/dev/null | shuf -n1)
          if [ -n "$TARGET" ]; then
            MOD=$((RANDOM % 3))
            case $MOD in
              0) dd if=/dev/urandom bs=1 count=$((RANDOM % 4096 + 512)) >> "$TARGET" 2>/dev/null ;;
              1) truncate -s "$((RANDOM % 8192 + 256))" "$TARGET" ;;
              2) dd if=/dev/urandom of="$TARGET" bs=1 count=$(random_size) 2>/dev/null ;;
            esac
          fi
        elif [ "$OP_RAND" -lt 90 ] || [ "$FILE_COUNT" -gt "$MAX_FILES" ]; then
          # DELETE: 20% of ops (or forced when over max-files)
          TARGET=$(find "$DATA_DIR" -type f 2>/dev/null | shuf -n1)
          [ -n "$TARGET" ] && rm -f "$TARGET"
        else
          # RENAME: 10% of ops
          TARGET=$(find "$DATA_DIR" -type f 2>/dev/null | shuf -n1)
          if [ -n "$TARGET" ]; then
            DIR=$(dirname "$TARGET")
            mv "$TARGET" "$DIR/renamed_$(date +%s%N)_$$" 2>/dev/null || true
          fi
        fi

        sleep "$DELAY"
      done
  - path: /etc/systemd/system/virtwork-backup-churn.service
    content: |
      [Unit]
      Description=Virtwork backup churn workload
      After=data.mount
      Requires=data.mount
      [Service]
      Type=simple
      Environment=CHURN_DATA_PATH=/data/churn
      Environment=CHURN_RATE=5
      Environment=CHURN_FILE_SIZE_MIN=1024
      Environment=CHURN_FILE_SIZE_MAX=10485760
      Environment=CHURN_MAX_FILES=10000
      Environment=CHURN_DIR_DEPTH=5
      ExecStart=/usr/local/bin/virtwork-backup-churn.sh
      Restart=always
      RestartSec=5
      [Install]
      WantedBy=multi-user.target
runcmd:
  - systemctl enable --now virtwork-backup-churn.service

Use Case

• Backup/DR partners (Velero, Trilio, Kasten by Veeam, Commvault, Cohesity): Need continuous filesystem mutation to validate incremental backup correctness. A static disk (like the disk workload produces) is easy to back up — the hard part is tracking which files changed between backup windows, correctly handling deleted files, and restoring a consistent point-in-time snapshot from a churning filesystem.
• Storage snapshot partners: Need filesystem-level changes happening during CSI VolumeSnapshot operations to validate crash-consistency guarantees. The mix of create/modify/delete/rename operations exercises the snapshot driver's ability to capture a consistent filesystem state.
• Data protection partners: Need realistic file lifecycle patterns (not just raw I/O) to validate deduplication efficiency, changed-block tracking accuracy, and restore integrity. The configurable directory depth and file size range produce a realistic filesystem tree.
• Compliance/audit partners: Need to validate that file-level audit trails (who changed what, when) remain accurate under sustained churn — important for regulated environments running VMs on OpenShift.

Additional Context

• This workload reuses the data disk pattern from the existing disk workload — DataVolumeTemplates(), ExtraVolumes(), and ExtraDisks() are already proven for attaching a CDI-provisioned persistent disk to a VM.
• The operation distribution (40% create, 30% modify, 20% delete, 10% rename) is tunable and intentionally biased toward creation early (to build up a filesystem) and deletion later (when max-files is reached). This prevents unbounded disk usage.
• The self-balancing max-files threshold means the workload runs indefinitely without filling the disk — deletions automatically increase when the file count exceeds the threshold. This is important for multi-day validation runs.
• At the default rate (5 ops/sec) with average file sizes around 1MB, this produces ~5MB/sec of filesystem mutation — enough to stress incremental backup tracking without overwhelming a 20Gi data disk within minutes.
• The variety of modification types (append, truncate, full rewrite) exercises different changed-block-tracking strategies that backup products use.

Start by locating the existing disk workload and its DataVolumeTemplates(), ExtraVolumes(), and ExtraDisks() implementations. Compare how that workload integrates persistent storage and cloud-init, then add the backup-churn script and systemd service described here. Done means the workload deploys on a single VM and continuously creates, modifies, deletes, and renames files on the data disk.