fix(bench): run_bench.sh hardcodes the run date and uses GNU-only grep -oP, failing on macOS
- Dominant language
- Rust
- Stars
- 467
- Forks
- 54
- Avg merge
- 4h 25m
- Merged PRs (30d)
- 310
Description
## Summary
`benchmarks/run_bench.sh` stamps a hardcoded run date, hardware string, and MLX version into every CSV row, and parses tokens with GNU-only `grep -oP`. On macOS (BSD grep) the `-P` match silently returns empty, so every row records as `FAILED`.
## Background
The script writes fixed `DATE`, `HARDWARE`, and `MLX_VERSION` values into the CSV regardless of when or where it runs, so recorded metadata is wrong on any host that is not the one the constants describe. Its token extraction relies on Perl-compatible regex, which BSD grep rejects, leaving `PREFILL_TOKS`/`DECODE_TOKS` empty and the row marked failed. The file also lacks `set -euo pipefail`, unlike the `scripts/*.sh` convention.
## Proposed Solution
- Set `DATE=$(date +%F)`.
- Make `HARDWARE` and `MLX_VERSION` environment-overridable with the current values as defaults.
- Replace `grep -oP` with a portable equivalent (`sed -E` or `awk`).
- Add `set -euo pipefail`.
## Implementation Notes
- `benchmarks/run_bench.sh:10-12` sets `DATE="2026-03-15"`, `HARDWARE="NVIDIA_GB10_CUDA13.0"`, `MLX_VERSION="0.31.1"`, written into the CSV rows.
- Lines 24 and 26 use `grep -oP '[\d.]+(?= tok/s)'`, the only `-P` usage under `scripts/` or `benchmarks/`.
- The file starts with `#!/bin/bash` and no `set` guard.
## Acceptance Criteria
- [ ] Rows carry the actual run date; hardware/version are overridable
- [ ] The script parses tokens correctly under BSD grep (macOS)
---
## Original Suggestion
### Title: fix(bench): run_bench.sh stamps a hardcoded date and its grep -oP silently fails on macOS
`benchmarks/run_bench.sh` stamps a hardcoded run date, hardware, and MLX version into every CSV row, and its token parsing uses GNU-only `grep -oP` — on macOS (BSD grep) that match silently comes back empty, so every row records as `FAILED`.
## Evidence
- `benchmarks/run_bench.sh:10-12`:
```bash
DATE="2026-03-15"
HARDWARE="NVIDIA_GB10_CUDA13.0"
MLX_VERSION="0.31.1"
```
written verbatim into the CSV at lines 31 and 34 regardless of when or where the script runs.
- Lines 25 and 27 use `grep -oP` — the only `-P` usage anywhere under `scripts/` or `benchmarks/`; BSD grep rejects `-P`, so `DECODE_TOKS` is empty on macOS
- The 34-line file also has no `set -euo pipefail`, unlike the `scripts/*.sh` convention
## Suggested fix
`DATE=$(date +%F)`; make `HARDWARE` and `MLX_VERSION` env-overridable with the current values as defaults; replace `grep -oP` with `sed -E` or `awk`; add `set -euo pipefail`.
## Acceptance criteria
- [ ] Rows carry the actual run date; hardware/version are overridable
- [ ] The script parses tokens correctly under BSD grep (macOS)
Contributor guide
Research direction
Start by reading benchmarks/run_bench.sh, especially lines 10-12, 24, and 26, then run it in a macOS/BSD grep environment to reproduce the empty token fields. Done means CSV rows use the current date, accept hardware and version overrides, and no longer become FAILED because token extraction is empty.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- bash
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100