IntersectMBO / IntersectMBO/plutus

Automated costing

Open
#4,384 3 comments 1 reaction 0 assignees View on GitHub
Low priority status: triaged
Dominant language
Haskell
Stars
1.6k
Forks
508
Avg merge
3d 10h
Merged PRs (30d)
22

Description

### Describe the feature you'd like

According to the information provided [here](https://github.com/input-output-hk/plutus/issues/4375) and [here](https://github.com/input-output-hk/plutus/blob/master/plutus-core/cost-model/CostModelGeneration.md), the current workflow for costing a new builtin is as follows:

1. Add benches for said builtin in `plutus-core/cost-model`. This step is unavoidably manual, but can't really be any other way.
2. Run `cabal bench plutus-core:cost-model-budgeting-bench`. This step is automatic, but _extremely_ slow; my attempt [here](https://github.com/input-output-hk/plutus/pull/4368#issuecomment-1031782137) took close to 12 hours to complete on _very_ nontrivially capable hardware.
3. Run `cabal bench plutus-core:update-cost-model`. This step is automatic, but with some caveats, as the linear regression being used might produce certain 'correct but nonsensical' outcomes, such as negative cost coefficients.
4. Rebuild to generate costing functions. This step is automatic.
5. Run `cabal bench plutus-core:cost-model-test` to verify that our costing makes sense. This step is automatic (enough).
6. Run more benchmarks for larger programs. This step is not automatic, but there's no reason why it couldn't be.

At MLabs, we believe it is important that costing reflects optimizations, ideally without requiring too much manual intervention. The current system does contain significant automation, but still requires not only considerable manual intervention, but isn't terribly practical, as evidenced by my attempt described above. It's not reasonable to run something this time-consuming as an automatic job every time someone (attempts to) optimize a primitive, or indeed, does something that might cause a primitive to _regress_, and require a more pessimistic costing. In an ideal world, this would be part of the CI: specifically, if we have a costing regression, CI would blow up.

Lastly, the 'reference machine' is a serious problem, as it 'ties' costing to a specific combination of hardware and OS. Additionally, it makes it impossible for contributors to do costing, although this is a stated future goal [here](https://github.com/input-output-hk/plutus/issues/4375#issuecomment-1029864939). In fact, I _already_ ran [afoul of this](https://github.com/input-output-hk/plutus/pull/4368#issuecomment-1032025802)!

### Describe alternatives you've considered

This is a tricky intersection of problems. On the one hand, in order to run the process of costing as part of CI, it would need to be significantly faster than it currently is. However, it's not clear how this would be possible: it would require investigation as to how the longest stage could be optimized. Furthermore, the question of the 'reference machine' needs to be addressed before we can do this.

One easy way to avoid the problem of a reference machine are 'yardstick primitives'. We essentially select some trusted primitives to act as a 'reference measure', or 'definition' of `1` and `n`, then measure everything by comparison to these. So thus, a cost would not be an absolute value: instead, it would be a function of the costs of one or more 'yardstick primitives'. This would solve the reference machine problem at least.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.