run_experiment takes longer in GCP when generating report
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 302
- PR merge metrics
- No merged PRs in 30d
Description
It seems like `run_experiment` in GCP takes longer to generate a report. It has been more than 5+ hours since I started executing following command:
```
PYTHONPATH=. python3 experiment/run_experiment.py \
--experiment-config experiment-config.yaml \
--benchmarks lcms_cms_transform_fuzzer \
--experiment-name $EXPERIMENT_NAME \
--fuzzers aflplusplus
```
for 2 trials, each of them lasting for 30 minutes, but it hasn’t finished generating report data. Running the same command and configuration in the local experiment already terminated.
The `run_experiment` is looping in this loop: https://github.com/google/fuzzbench/blob/ba22647276a63912312fb6636287cc266da19683/experiment/dispatcher.py#L166.
It seems like the processes are sleeping for the majority of time. `htop` output also shows that measuring processes are getting scheduled, but there’s less than 1% usage of CPU for each scheduled process.
Is there any solution to make it generate report faster?
`run_experiment` script also steadily outputs this warning in case it is relevant.
```
/usr/local/lib/python3.10/site-packages/jinja2/runtime.py:298: FutureWarning: this method is deprecated in favour of `Styler.to_html()`
return __obj(*args, **kwargs)
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
in the loop DEBUG
/usr/local/lib/python3.10/site-packages/jinja2/runtime.py:298: FutureWarning: this method is deprecated in favour of `Styler.to_html()`
return __obj(*args, **kwargs)
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/work/src/analysis/plotting.py:159: FutureWarning:
The `ci` parameter is deprecated. Use `errorbar=('ci', 95)` for the same effect.
axes = sns.lineplot(
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:202: RuntimeWarning: All-NaN slice encountered
vmin = np.nanmin(calc_data)
/usr/local/lib/python3.10/site-packages/seaborn/matrix.py:207: RuntimeWarning: All-NaN slice encountered
vmax = np.nanmax(calc_data)
```
Contributor guide
Assessment
This issue has not been assessed yet.