Not able to cross 2000 Samples/second for RGAT inferencing on tiny dataset

Hello All!

Please help me with the following issue:

I am carrying out inferencing for RGAT model on tiny dataset with the following values of --threads and --max-batchsize:

THREAD_MIN=4
THREAD_MAX=224
THREAD_STEP=4
BATCH_MIN=10
BATCH_MAX=4096
BATCH_STEP=10

Here is the complete command:

#!/usr/bin/env bash
set -u

# ==== CONFIG (edit if you like) ====
export OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
COUNT=40000000        # big count so each run stabilizes
LAYOUTS=(CSC CSR)     # add COO if you want
THREAD_MIN=4
THREAD_MAX=224
THREAD_STEP=4
BATCH_MIN=10
BATCH_MAX=4096
BATCH_STEP=10

# ==== OUTPUT ====
mkdir -p output
SUMMARY_CSV="output/throughput_summary.csv"
TOP_TXT="output/throughput_top.txt"
echo "layout,threads,batch,count,samples_per_sec,output_dir" > "$SUMMARY_CSV"

# ==== SWEEP ====
for layout in "${LAYOUTS[@]}"; do
  for t in $(seq "$THREAD_MIN" "$THREAD_STEP" "$THREAD_MAX"); do
    for b in $(seq "$BATCH_MIN" "$BATCH_STEP" "$BATCH_MAX"); do
      out="output/${layout}_t${t}_b${b}"
      rm -rf "$out" && mkdir -p "$out"
      echo "Running: layout=${layout} threads=${t} batch=${b} count=${COUNT}"

      # Run; if a combo fails/OOMs, keep going
      python3 main.py \
        --dataset igbh-dgl-tiny --dataset-path igbh/ \
        --model-path model/RGAT.pt \
        --device gpu --dtype fp32 \
        --scenario Offline --layout "${layout}" \
        --count "${COUNT}" \
        --max-batchsize "${b}" --threads "${t}" \
        --output "$out" --user_conf user.conf --audit_conf audit.config || true

      # Extract throughput
      if [[ -f "$out/mlperf_log_summary.txt" ]]; then
        sps=$(grep -E "Samples per second|Completed samples per second" "$out/mlperf_log_summary.txt" \
              | tail -1 | awk -F': ' '{print $2}' | tr -d '[:space:]')
      else
        sps=""
      fi
      [[ -z "$sps" ]] && sps="NA"

      printf "%s,%s,%s,%s,%s,%s\n" "$layout" "$t" "$b" "$COUNT" "$sps" "$out" >> "$SUMMARY_CSV"
      echo "=> ${layout} t=${t} b=${b} : ${sps} samples/s"
    done
  done
done

# ==== RANK RESULTS (highest first) ====
{
  head -1 "$SUMMARY_CSV"
  tail -n +2 "$SUMMARY_CSV" | awk -F, '$5 != "NA"' | sort -t, -k5,5gr
  tail -n +2 "$SUMMARY_CSV" | awk -F, '$5 == "NA"'
} | tee "$TOP_TXT"

echo
echo "Top 10:"
awk -F, 'NR==1{print;next} $5!="NA"{print}' "$SUMMARY_CSV" | sort -t, -k5,5gr | head -10 | column -t -s,
echo
echo "Saved:"
echo "  CSV: $SUMMARY_CSV"
echo "  Sorted view: $TOP_TXT"

Please tell me if I am missing out on something.

I aslo see my 8 H200s underutilised, as only 1 GPU is used.

Thanks

Start with the main.py invocation and the supplied user.conf and audit.config, then inspect the generated mlperf_log_summary.txt files and throughput_summary.csv for failed runs and GPU utilization. Determine whether the reported single-GPU use and throughput result from the benchmark configuration or an implementation limitation; done means documenting a reproducible cause or the missing configuration needed to use all eight H200s.