[Workload]: log-generator

Workload Name

log-generator

Workload Description

Structured log emission workload that generates logs at configurable rates and formats. Produces sustained log volume in common formats — syslog (RFC 5424), JSON (structured), Apache combined, and nginx access log — written to journald and/or log files.

This workload fills the gap for log management and SIEM partners. The current virtwork workloads produce system-level metrics (CPU, memory, disk I/O) but generate minimal log output. Partners building log aggregation, analysis, and security information products (Splunk, Elastic, Datadog Logs, Sumo Logic, CrowdStrike Falcon LogScale) need sustained log volume with realistic structure and variety to validate their agents correctly collect, parse, and forward logs from VMs on OpenShift.

Tooling and Packages

• Tool: flog (fake log generator — supports Apache combined, RFC5424, JSON, Common Log Format) or a lightweight shell script using logger (for syslog) and structured echo loops (for JSON/file-based)
• RPM packages: none required if using a shell-based generator; flog is a single Go binary
• systemd service command: flog -f json -d 100ms -l (generate JSON logs at 10 lines/sec, infinite loop)
• Configurable parameters: format (syslog, json, apache, nginx), rate (lines/sec), output target (stdout→journald, file path), log line size, number of unique source IPs/paths (controls cardinality)

VM Count Model

Single VM (like cpu, memory, disk)

Required Resources

• Persistent storage (DataVolume)
• Kubernetes Service (for inter-VM communication)
• Kubernetes Secret (for credentials or config)
• Additional CPU/memory beyond defaults
• GPU or special device passthrough

Cloud-Init Details

write_files:
  - path: /usr/local/bin/virtwork-log-generator.sh
    permissions: '0755'
    content: |
      #!/bin/bash
      # Configurable log generation
      FORMAT="${LOG_FORMAT:-json}"
      RATE="${LOG_RATE:-10}"  # lines per second
      DELAY=$(echo "scale=3; 1/$RATE" | bc)
      while true; do
        case "$FORMAT" in
          json)
            echo "{\"timestamp\":\"$(date -Iseconds)\",\"level\":\"$(shuf -n1 -e INFO WARN ERROR DEBUG)\",\"service\":\"virtwork-app\",\"message\":\"$(shuf -n1 -e 'Request processed' 'Connection established' 'Cache miss' 'Query completed' 'Auth validated')\",\"duration_ms\":$((RANDOM % 500)),\"status\":$(shuf -n1 -e 200 201 301 400 404 500)}"
            ;;
          syslog)
            logger -p "$(shuf -n1 -e local0.info local0.warn local0.err)" "virtwork-app: $(shuf -n1 -e 'request_handled' 'connection_timeout' 'cache_refreshed' 'query_slow') duration=$((RANDOM % 500))ms"
            ;;
          apache)
            echo "$(printf "%d.%d.%d.%d" $((RANDOM%256)) $((RANDOM%256)) $((RANDOM%256)) $((RANDOM%256))) - - [$(date '+%d/%b/%Y:%H:%M:%S %z')] \"GET /$(shuf -n1 -e api/v1/users api/v1/orders health metrics static/app.js) HTTP/1.1\" $(shuf -n1 -e 200 200 200 301 404 500) $((RANDOM % 50000))"
            ;;
        esac
        sleep "$DELAY"
      done
  - path: /etc/systemd/system/virtwork-log-generator.service
    content: |
      [Unit]
      Description=Virtwork log generation workload
      After=network-online.target
      [Service]
      Type=simple
      Environment=LOG_FORMAT=json
      Environment=LOG_RATE=10
      ExecStart=/usr/local/bin/virtwork-log-generator.sh
      Restart=always
      RestartSec=5
      StandardOutput=journal
      SyslogIdentifier=virtwork-log-generator
      [Install]
      WantedBy=multi-user.target
runcmd:
  - systemctl enable --now virtwork-log-generator.service

Use Case

• Log management partners (Splunk, Elastic, Datadog Logs): Need sustained log volume from VMs to validate their agents correctly discover, collect, parse, and forward logs. Different formats test different parsing pipelines.
• SIEM partners (CrowdStrike, Splunk ES, IBM QRadar): Need structured security-relevant log events (auth failures, connection anomalies) to validate detection rules and correlation engines against VM-sourced logs on OpenShift.
• Observability partners: Need correlated logs alongside the metrics generated by other virtwork workloads (CPU, memory, disk) to validate unified observability dashboards.
• Compliance/audit partners: Need to validate that log collection from VMs meets retention and completeness requirements — log-generator with known output rate provides a ground truth for verifying no logs are dropped.

Additional Context

• The shell-based approach is preferred for initial implementation: zero external dependencies, easy to extend formats, and the script itself is the cloud-init payload. flog can be offered as an alternative for higher throughput.
• Configurable parameters to expose via CLI/YAML: log-format (json, syslog, apache, nginx), log-rate (lines/sec), log-output (journal, file, both), log-line-size (bytes), log-cardinality (number of unique source IPs/paths/services — controls how many unique log streams the partner product must handle).
• Output to journald (via StandardOutput=journal) ensures logs are available through the standard Linux logging pipeline that most partner agents hook into. File output can target /var/log/virtwork/ for agents that tail files.
• At 100 lines/sec with ~200 byte JSON lines, this generates ~1.7 MB/min or ~2.4 GB/day per VM — realistic for a moderately active application server.

Review the existing cpu, memory, and disk workloads first to find how VM workloads and cloud-init payloads are represented. The proposal names /usr/local/bin/virtwork-log-generator.sh and virtwork-log-generator.service as the main entry points, with journald and optional file output. Done means a single-VM workload can generate the agreed formats and configurable rate and output targets reliably.