argotorg / argotorg/solidity

Refine enforce gas cost min value threshold behaviour in ``isoltest``

Open
#12,527 1 comment 0 reactions 0 assignees View on GitHub
low effort medium impact must have eventually testing :hammer:
Dominant language
C++
Stars
25.7k
Forks
6.2k
Avg merge
2d 19h
Merged PRs (30d)
29

Description

Currently ``isoltest`` just ignores any existing test expectations, if the actually consumed gas amount is below the ``--enforce-gas-cost-min-value`` value.
For example ``test/libsolidity/semanticTests/externalContracts/snark.sol`` at the time of writing says ``// gas irOptimized: 95261`` for ``verifyTx()``. The actual value currently is ``95259``, but the test passes.
The test also passes if I change the expectation to any arbitrary value (both below or above the "enforce-gas-cost-min-value" threshold).

This behaviour is not particularly nice - it will lead to ghost-test-expectations whenever a case falls below the gas threshold.

The only thing determined by ``--enforce-gas-cost-min-value`` should be to determine at which threshold ``isoltest`` should start to actively *add* gas expectations. Existing expectations should always be checked and kept up to date. [alternative: actively *remove* expectations below the threshold; I prefer always checking, though]

IIRC this was always the intended meaning of the option, but as @cameel pointed out, it's name is not particularly well chosen, so in the process it might make sense to rename it.

Contributor guide

Open the contributing guide

Research direction

Start with isoltest's handling of --enforce-gas-cost-min-value and reproduce the behavior using test/libsolidity/semanticTests/externalContracts/snark.sol, especially verifyTx(). Check that existing gas expectations are still validated below the threshold, while the option controls only when expectations are added; done means the arbitrary 95259/95261 expectation case no longer passes silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, solidity
Domain
compilers, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.