automl / automl/NASLib

Source of validation accuracy in zero-cost case

Open
#133 2 comments 0 reactions 2 assignees Claimed by @abhash-er View on GitHub
zero cost merge
Dominant language
Python
Stars
595
Forks
133
PR merge metrics
No merged PRs in 30d

Description

In the zero-cost branch optimizers `Npenas` and `Bananas`, the validation accuracy of architectures is being queried from the zero-cost-benchmark as follows:

```python
model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']
```

The question is, whether this supports the case where the user wants to use the `ZeroCost` predictor because their dataset or search space is not supported by the zero-cost benchmark.

If this is a case that we want to support, one option would be to introduce a parameter `use_zc_api` and use it as follows:

```python
if self.use_zc_api:
model.accuracy = self.zc_api[str(model.arch_hash)]['val_accuracy']
else:
model.accuracy = model.arch.query(
self.performance_metric, self.dataset, dataset_api=self.dataset_api
)
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.