lablup / lablup/mlxcel

fix(bench): run_bench.sh hardcodes the run date and uses GNU-only grep -oP, failing on macOS

Open Beginner friendly
#1,666 0 comments 0 reactions 0 assignees View on GitHub
priority:medium status:ready type:bug
Dominant language
Rust
Stars
467
Forks
54
Avg merge
4h 25m
Merged PRs (30d)
310

Description

## Summary

`benchmarks/run_bench.sh` stamps a hardcoded run date, hardware string, and MLX version into every CSV row, and parses tokens with GNU-only `grep -oP`. On macOS (BSD grep) the `-P` match silently returns empty, so every row records as `FAILED`.

## Background

The script writes fixed `DATE`, `HARDWARE`, and `MLX_VERSION` values into the CSV regardless of when or where it runs, so recorded metadata is wrong on any host that is not the one the constants describe. Its token extraction relies on Perl-compatible regex, which BSD grep rejects, leaving `PREFILL_TOKS`/`DECODE_TOKS` empty and the row marked failed. The file also lacks `set -euo pipefail`, unlike the `scripts/*.sh` convention.

## Proposed Solution

- Set `DATE=$(date +%F)`.
- Make `HARDWARE` and `MLX_VERSION` environment-overridable with the current values as defaults.
- Replace `grep -oP` with a portable equivalent (`sed -E` or `awk`).
- Add `set -euo pipefail`.

## Implementation Notes

- `benchmarks/run_bench.sh:10-12` sets `DATE="2026-03-15"`, `HARDWARE="NVIDIA_GB10_CUDA13.0"`, `MLX_VERSION="0.31.1"`, written into the CSV rows.
- Lines 24 and 26 use `grep -oP '[\d.]+(?= tok/s)'`, the only `-P` usage under `scripts/` or `benchmarks/`.
- The file starts with `#!/bin/bash` and no `set` guard.

## Acceptance Criteria

- [ ] Rows carry the actual run date; hardware/version are overridable
- [ ] The script parses tokens correctly under BSD grep (macOS)

---

## Original Suggestion

### Title: fix(bench): run_bench.sh stamps a hardcoded date and its grep -oP silently fails on macOS

`benchmarks/run_bench.sh` stamps a hardcoded run date, hardware, and MLX version into every CSV row, and its token parsing uses GNU-only `grep -oP` — on macOS (BSD grep) that match silently comes back empty, so every row records as `FAILED`.

## Evidence

- `benchmarks/run_bench.sh:10-12`:

```bash
DATE="2026-03-15"
HARDWARE="NVIDIA_GB10_CUDA13.0"
MLX_VERSION="0.31.1"
```

written verbatim into the CSV at lines 31 and 34 regardless of when or where the script runs.

- Lines 25 and 27 use `grep -oP` — the only `-P` usage anywhere under `scripts/` or `benchmarks/`; BSD grep rejects `-P`, so `DECODE_TOKS` is empty on macOS
- The 34-line file also has no `set -euo pipefail`, unlike the `scripts/*.sh` convention

## Suggested fix

`DATE=$(date +%F)`; make `HARDWARE` and `MLX_VERSION` env-overridable with the current values as defaults; replace `grep -oP` with `sed -E` or `awk`; add `set -euo pipefail`.

## Acceptance criteria

- [ ] Rows carry the actual run date; hardware/version are overridable
- [ ] The script parses tokens correctly under BSD grep (macOS)

Contributor guide

Open the contributing guide

Research direction

Start by reading benchmarks/run_bench.sh, especially lines 10-12, 24, and 26, then run it in a macOS/BSD grep environment to reproduce the empty token fields. Done means CSV rows use the current date, accept hardware and version overrides, and no longer become FAILED because token extraction is empty.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
performance, tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.