High memory usage (~14GB) when exporting large datasets (e.g. 2 million rows, ~700MB file)

backend/runner/taskrun/data_export_executor.go の DoExport から始めて exportCSV を追跡し、issue に記載されている結果セット全体の実体化と bytes.Buffer のアロケーションに注目します。issue で報告されている 2-million-row、~700 MB のデータセットのような大規模なエクスポートをプロファイリングし、メモリ内バッファリングを最小限にしてエクスポート後にデータを解放することで、メモリ使用量が削減されることを確認します。

Is your feature request related to a problem?

Yes. When using the Export Center to export a large dataset (around 2 million rows, ~700MB file), Bytebase’s memory usage spikes to around 14GB.

Describe the solution you'd like

I hope the export mechanism can be optimized to use memory more efficiently.
During large data exports, Bytebase should minimize in-memory buffering and promptly release memory once the export is finished.

Additional context

Environment:

• Bytebase Version: 3.11.0
• Database: MySQL 5.7.30
• Deployment Binary: Mac

Observed memory usage: ~14GB during export. mem.log
Suggest adding monitoring or log tracing for export performance.

• export data list

-rw-r--r--@ 1 mako  staff   689M Oct 28 15:30 test.csv
-rw-r--r--@ 1 mako  staff    41M Oct 28 15:30 export-data-2025-10-28T15-30-38.zip

• monitor shell

while ps -p "$PID" >/dev/null 2>&1; do
	ts=$(date '+%Y-%m-%d %H:%M:%S')

	line=$(top -l 1 -pid $PID -stats pid,cpu,mem | awk -v pid=$PID '$1==pid {print}')
	if [ -z "$line" ]; then
		sleep 2
		continue
	fi

	pid_val=$(echo $line | awk '{print $1}')
	cpu_val=$(echo $line | awk '{print $2}')
	mem_str=$(echo $line | awk '{print $3}')

	if [[ $mem_str == *G ]]; then
		mem_val=$(awk -v m="${mem_str%G}" 'BEGIN{printf "%.1f", m*1024}')
	elif [[ $mem_str == *M ]]; then
		mem_val=$(awk -v m="${mem_str%M}" 'BEGIN{printf "%.1f", m}')
	else
		mem_val=0
	fi

	output="$ts $pid_val $cpu_val $mem_val"
	echo "$output"

	sleep 2
done

Below is a memory usage analysis captured during one large export task (~2 million rows, ~700 MB data):

• Key Observations
  DoExport

• Detailed Findings
  • Result set fully materialized: the entire query result is held in memory (v1pb.QueryResult → Rows → Values), including desensitized copies.
  • Encoding stage is the main source of memory growth: exportCSV constructs the full file in memory using bytes.Buffer, repeatedly expanding and copying data.
  • Compression/encryption overhead is minimal in comparison.
  • GC didn’t trigger during CSV encoding, likely because large objects remained referenced.
  • In XLSX exports (not sampled here), memory use is typically even higher, since excelize keeps a full workbook model plus a final ZIP buffer in memory simultaneously.