Call for new benchmarks

Call for new benchmarks

The goal is to develop, test, deploy and evaluate Machine Learning benchmarks on a wide range of computing platforms. These benchmarks will be a starting point for exploring new ML methods, they can be used for ranking computers and help towards a better understanding of the interaction between ML applications and the underlying hardware. The main components of each benchmark are the scientific value, dataset, implementation and documentation. Details on how to contribute a benchmark can be found in the Policy and Submission Rules documents.

In the current suite there are four benchmarks which are drawn from various scientific domains such as material, life and earth sciences. The benchmarks are written in Python and they use libraries such as TensorFlow and PyTorch. The datasets are coming from various sources which include meteorological satellites, scanning tunnelling microscopes, seismographs and DNA sequencers. The size of these datasets covers a range from tens to hundreds of GigaBytes.

Start by reading policy.adoc and submit.adoc, then inspect the existing entries in benchmarks. A contribution would need a scientific value, dataset, implementation, and documentation, but this issue does not identify a specific benchmark or entry point to change.