NVIDIA / NVIDIA/cuvs

Lucene: Memory Consumption Discrepancy between CAGRA on GPU and HNSW on CPU resolved

Open
#2,451 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Lucene
Dominant language
Cuda
Stars
854
Forks
236
Avg merge
3d 3h
Merged PRs (30d)
62

Description

I ran comparable suites of Pareto parameter-grid-search benchmarks on GPU and CPU and found that peak RAM during CPU benchmarks was 19.27 GB, while peak RAM during GPU benchmarks was 18.35 GB.

I wrote two scripts, one for CPU and one for GPU, that poll their ec2 instance for RAM usage every second during the tests and then outputs the results to a file that I can do a linear scan for peak usage.

for reference, here are my scripts:

gpu_wrapper.sh:
<gpu_wrapper>

#!/bin/bash

# Configuration
MEMORY_THRESHOLD=80  # Alert when system memory usage exceeds this %
GPU_THRESHOLD=80     # Alert when GPU memory usage exceeds this %
SAMPLE_INTERVAL=1    # Sample every 1 second
LOG_FILE="memory_monitor.log"
SPIKE_LOG="memory_spikes.log"

# Initialize tracking variables
START_TIME=$(date +%s)
MAX_RAM_USAGE=0
MAX_GPU_USAGE=0
SPIKE_COUNT=0

# Check if GPU is available
HAS_GPU=false
if command -v nvidia-smi &> /dev/null; then
    if nvidia-smi &> /dev/null; then
        HAS_GPU=true
    fi
fi

# Clear previous logs
> "$LOG_FILE"
> "$SPIKE_LOG"

# Function to get current memory usage percentage
get_ram_usage() {
    free | awk '/Mem:/ {printf "%.1f", $3/$2 * 100}'
}

# Function to get GPU memory usage percentage
get_gpu_usage() {
    if [ "$HAS_GPU" = true ]; then
        nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits | \
        awk -F', ' '{printf "%.1f", ($1/$2)*100}'
    else
        echo "0"
    fi
}

# Function to compare floating point numbers (returns 1 if $1 > $2, else 0)
float_gt() {
    awk -v n1="$1" -v n2="$2" 'BEGIN {if (n1 > n2) print 1; else print 0}'
}
# Function to log and alert on spikes
check_spike() {
    local ram_usage=$1
    local gpu_usage=$2
    local timestamp=$3
    local elapsed=$4
    
    # Check RAM spike
    if [ "$(float_gt "$ram_usage" "$MEMORY_THRESHOLD")" -eq 1 ]; then
        echo "[SPIKE] Time: ${elapsed}s | RAM: ${ram_usage}% (threshold: ${MEMORY_THRESHOLD}%)"
        echo "$(date '+%Y-%m-%d %H:%M:%S') | Elapsed: ${elapsed}s | RAM SPIKE: ${ram_usage}%" >> "$SPIKE_LOG"
        ((SPIKE_COUNT++))
    fi
    
    # Check GPU spike only if GPU is available
    if [ "$HAS_GPU" = true ]; then
        if [ "$(float_gt "$gpu_usage" "$GPU_THRESHOLD")" -eq 1 ]; then
            echo "[SPIKE] Time: ${elapsed}s | GPU: ${gpu_usage}% (threshold: ${GPU_THRESHOLD}%)"
            echo "$(date '+%Y-%m-%d %H:%M:%S') | Elapsed: ${elapsed}s | GPU SPIKE: ${gpu_usage}%" >> "$SPIKE_LOG"
            ((SPIKE_COUNT++))
        fi
    fi
}

# Monitoring function that runs in background
monitor_memory() {
    while kill -0 $MAIN_PID 2>/dev/null; do
        CURRENT_TIME=$(date +%s)
        ELAPSED=$((CURRENT_TIME - START_TIME))
        
        RAM_USAGE=$(get_ram_usage)
        GPU_USAGE=$(get_gpu_usage)
        
        # Log to file
        if [ "$HAS_GPU" = true ]; then
            echo "${ELAPSED},${RAM_USAGE},${GPU_USAGE}" >> "$LOG_FILE"
        else
            echo "${ELAPSED},${RAM_USAGE},N/A" >> "$LOG_FILE"
        fi
        
        # Track peak usage
        if [ "$(float_gt "$RAM_USAGE" "$MAX_RAM_USAGE")" -eq 1 ]; then
            MAX_RAM_USAGE=$RAM_USAGE
        fi
        
        if [ "$HAS_GPU" = true ]; then
            if [ "$(float_gt "$GPU_USAGE" "$MAX_GPU_USAGE")" -eq 1 ]; then
                MAX_GPU_USAGE=$GPU_USAGE
            fi
        fi
        # Check for spikes and print in real-time
        check_spike "$RAM_USAGE" "$GPU_USAGE" "$(date '+%Y-%m-%d %H:%M:%S')" "$ELAPSED"
        
        sleep $SAMPLE_INTERVAL
    done
}

echo "=========================================="
echo "Memory Monitoring Wrapper"
echo "Started at: $(date '+%Y-%m-%d %H:%M:%S')"
if [ "$HAS_GPU" = true ]; then
    echo "GPU detected: Monitoring both RAM and GPU memory"
    echo "GPU Threshold: ${GPU_THRESHOLD}%"
else
    echo "No GPU detected: Monitoring RAM only"
fi
echo "RAM Threshold: ${MEMORY_THRESHOLD}%"
echo "=========================================="
echo ""

# Start the main script in background
./run_sweep.sh \
    --data-dir /workspace/datasets \
    --datasets datasets.json \
    --sweeps sweeps.json \
    --configs-dir configs \
    --results-dir results \
    --run-benchmarks &

MAIN_PID=$!
echo "Started run_sweep.sh with PID: $MAIN_PID"
echo ""
# Start monitoring in background
monitor_memory &
MONITOR_PID=$!

# Wait for main script to complete
wait $MAIN_PID
MAIN_EXIT_CODE=$?

# Stop monitoring
kill $MONITOR_PID 2>/dev/null
wait $MONITOR_PID 2>/dev/null

# Calculate total time
END_TIME=$(date +%s)
TOTAL_TIME=$((END_TIME - START_TIME))
HOURS=$((TOTAL_TIME / 3600))
MINUTES=$(((TOTAL_TIME % 3600) / 60))
SECONDS=$((TOTAL_TIME % 60))

# Print summary
echo ""
echo "=========================================="
echo "Execution Summary"
echo "=========================================="
echo "Completed at: $(date '+%Y-%m-%d %H:%M:%S')"
echo "Total Running Time: ${HOURS}h ${MINUTES}m ${SECONDS}s"
echo "Exit Code: $MAIN_EXIT_CODE"
echo ""
echo "Memory Statistics:"
echo "  Peak RAM Usage: ${MAX_RAM_USAGE}%"
if [ "$HAS_GPU" = true ]; then
    echo "  Peak GPU Usage: ${MAX_GPU_USAGE}%"
fi
echo "  Total Spikes Detected: $SPIKE_COUNT"
echo ""

if [ -s "$SPIKE_LOG" ]; then
    echo "Memory Spikes:"
    cat "$SPIKE_LOG"
    echo ""
fi

echo "Detailed logs saved to: $LOG_FILE"
echo "Spike logs saved to: $SPIKE_LOG"
echo "=========================================="

exit $MAIN_EXIT_CODE

</gpu_wrapper>

cpu_wrapper.sh:
<cpu_wrapper>

#!/bin/bash

# Configuration
MEMORY_THRESHOLD=80  # Alert when system memory usage exceeds this %
GPU_THRESHOLD=80     # Alert when GPU memory usage exceeds this %
SAMPLE_INTERVAL=1    # Sample every 1 second
LOG_FILE="memory_monitor.log"
SPIKE_LOG="memory_spikes.log"

# Initialize tracking variables
START_TIME=$(date +%s)
MAX_RAM_USAGE=0
MAX_GPU_USAGE=0
SPIKE_COUNT=0

# Check if GPU is available
HAS_GPU=false
if command -v nvidia-smi &> /dev/null; then
    if nvidia-smi &> /dev/null; then
        HAS_GPU=true
    fi
fi

# Clear previous logs
> "$LOG_FILE"
> "$SPIKE_LOG"

# Function to get current memory usage percentage
get_ram_usage() {
    free | awk '/Mem:/ {printf "%.1f", $3/$2 * 100}'
}

# Function to get GPU memory usage percentage
get_gpu_usage() {
    if [ "$HAS_GPU" = true ]; then
        nvidia-smi --query-gpu=memory.used,memory.total --format=csv,noheader,nounits | \
        awk -F', ' '{printf "%.1f", ($1/$2)*100}'
    else
        echo "0"
    fi
}
# Function to compare floating point numbers (returns 1 if $1 > $2, else 0)
float_gt() {
    awk -v n1="$1" -v n2="$2" 'BEGIN {if (n1 > n2) print 1; else print 0}'
}
# Function to log and alert on spikes
check_spike() {
    local ram_usage=$1
    local gpu_usage=$2
    local timestamp=$3
    local elapsed=$4
    
    # Check RAM spike
    if [ "$(float_gt "$ram_usage" "$MEMORY_THRESHOLD")" -eq 1 ]; then
        echo "[SPIKE] Time: ${elapsed}s | RAM: ${ram_usage}% (threshold: ${MEMORY_THRESHOLD}%)"
        echo "$(date '+%Y-%m-%d %H:%M:%S') | Elapsed: ${elapsed}s | RAM SPIKE: ${ram_usage}%" >> "$SPIKE_LOG"
        ((SPIKE_COUNT++))
    fi
    
    # Check GPU spike only if GPU is available
    if [ "$HAS_GPU" = true ]; then
        if [ "$(float_gt "$gpu_usage" "$GPU_THRESHOLD")" -eq 1 ]; then
            echo "[SPIKE] Time: ${elapsed}s | GPU: ${gpu_usage}% (threshold: ${GPU_THRESHOLD}%)"
            echo "$(date '+%Y-%m-%d %H:%M:%S') | Elapsed: ${elapsed}s | GPU SPIKE: ${gpu_usage}%" >> "$SPIKE_LOG"
            ((SPIKE_COUNT++))
        fi
    fi
}

# Monitoring function that runs in background
monitor_memory() {
    while kill -0 $MAIN_PID 2>/dev/null; do
        CURRENT_TIME=$(date +%s)
        ELAPSED=$((CURRENT_TIME - START_TIME))
        
        RAM_USAGE=$(get_ram_usage)
        GPU_USAGE=$(get_gpu_usage)
        
        # Log to file
        if [ "$HAS_GPU" = true ]; then
            echo "${ELAPSED},${RAM_USAGE},${GPU_USAGE}" >> "$LOG_FILE"
        else
            echo "${ELAPSED},${RAM_USAGE},N/A" >> "$LOG_FILE"
        fi
        
        # Track peak usage
        if [ "$(float_gt "$RAM_USAGE" "$MAX_RAM_USAGE")" -eq 1 ]; then
            MAX_RAM_USAGE=$RAM_USAGE
        fi
        
        if [ "$HAS_GPU" = true ]; then
            if [ "$(float_gt "$GPU_USAGE" "$MAX_GPU_USAGE")" -eq 1 ]; then
                MAX_GPU_USAGE=$GPU_USAGE
            fi
        fi
        
        # Check for spikes and print in real-time
        check_spike "$RAM_USAGE" "$GPU_USAGE" "$(date '+%Y-%m-%d %H:%M:%S')" "$ELAPSED"
        
        sleep $SAMPLE_INTERVAL
    done
}

echo "=========================================="
echo "Memory Monitoring Wrapper"
echo "Started at: $(date '+%Y-%m-%d %H:%M:%S')"
if [ "$HAS_GPU" = true ]; then
    echo "GPU detected: Monitoring both RAM and GPU memory"
    echo "GPU Threshold: ${GPU_THRESHOLD}%"
else
    echo "No GPU detected: Monitoring RAM only"
fi
echo "RAM Threshold: ${MEMORY_THRESHOLD}%"
echo "=========================================="
echo ""

# Start the main script in background
./run_sweep.sh \
    --data-dir /home/ec2-user/datasets \
    --datasets datasets.json \
    --sweeps sweeps.json \
    --configs-dir configs \
    --results-dir results \
    --run-benchmarks &

MAIN_PID=$!
echo "Started run_sweep.sh with PID: $MAIN_PID"
echo ""

# Start monitoring in background
monitor_memory &
MONITOR_PID=$!

# Wait for main script to complete
wait $MAIN_PID
MAIN_EXIT_CODE=$?

# Stop monitoring
kill $MONITOR_PID 2>/dev/null
wait $MONITOR_PID 2>/dev/null

# Calculate total time
END_TIME=$(date +%s)
TOTAL_TIME=$((END_TIME - START_TIME))
HOURS=$((TOTAL_TIME / 3600))
MINUTES=$(((TOTAL_TIME % 3600) / 60))
SECONDS=$((TOTAL_TIME % 60))

# Print summary
echo ""
echo "=========================================="
echo "Execution Summary"
echo "=========================================="
echo "Completed at: $(date '+%Y-%m-%d %H:%M:%S')"
echo "Total Running Time: ${HOURS}h ${MINUTES}m ${SECONDS}s"
echo "Exit Code: $MAIN_EXIT_CODE"
echo ""
echo "Memory Statistics:"
echo "  Peak RAM Usage: ${MAX_RAM_USAGE}%"
if [ "$HAS_GPU" = true ]; then
    echo "  Peak GPU Usage: ${MAX_GPU_USAGE}%"
fi
echo "  Total Spikes Detected: $SPIKE_COUNT"
echo ""

if [ -s "$SPIKE_LOG" ]; then
    echo "Memory Spikes:"
    cat "$SPIKE_LOG"
    echo ""
fi

echo "Detailed logs saved to: $LOG_FILE"
echo "Spike logs saved to: $SPIKE_LOG"
echo "=========================================="

exit $MAIN_EXIT_CODE

</cpu_wrapper>

get_peak.sh:
<get_peak>

#!/bin/bash

TOTAL_RAM=$(free -g | awk '/Mem:/ {print $2}')
awk -F',' -v total=$TOTAL_RAM '{gb=($2/100)*total; if(gb>max) {max=gb; time=$1}} END {printf "Peak RAM Usage: %.2f GB at %d seconds\n", max, time}' memory_monitor.log

</get_peak>

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with gpu_wrapper.sh, cpu_wrapper.sh, and get_peak.sh, then inspect how each invokes run_sweep.sh and calculates peak RAM from memory_monitor.log. Reproduce the comparable CPU and GPU benchmark runs using the shown dataset, sweep, config, and results paths. Done means explaining or correcting the reported peak-memory discrepancy with repeatable measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.