oasisprotocol / oasisprotocol/cli

cli: Make example tests smarter

Open
#328 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ci tests
Dominant language
Go
Stars
80
Forks
23
Avg merge
9h 43m
Merged PRs (30d)
3

Description

We have way too much .static examples meaning the output is not actually tested inside CI. I suggest we get rid of using .static for content that changes often and try to make a "smarter" diff.

For example, recent regression caused that ParaTime names got lost:

$ oasis network status
=== NETWORK STATUS ===
Network:      localhost_testnet
Node's ID:    5MsgQwijUlpH9+0Hbyors5jwmx7tTmKMA4c9leV3prI=
Core version: 24.3

==== Consensus ====
Status:               syncing
Version:              7.0.0
Chain context:        0b91b8e4e44b2003a7c5e23ddadb5e14ef5345c0ebcb3ddcae07fa2f244cab76
Latest height:        24115729 (2024-11-28 08:29:38 +0100 CET)
Latest block hash:    b7a9ad2fe687b9b2887512b03707347b0ec743020a175bc45c09979e3cc90d82
Latest epoch:         40176
Is validator:         false
Registration:         false

==== ParaTimes ====
  Status:               waiting for consensus sync
  Latest round:         7933058 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      23
  Status:               waiting for consensus sync
  Latest round:         8930789 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      22
  Status:               waiting for consensus sync
  Latest round:         9219372 (2024-11-28 08:29:38 +0100 CET)
  Last finalized round: 18446744073709551615
  Storage status:       initializing
  Number of peers:      23

We should ignore the latest height, block hash and epoch in this case, but still test everything else.

Perhaps a kind of a regex "mask" per example that should define how the output should look like. So when you run "make examples" and the regex is matched, then don't generate the new .out files.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the example definitions used by make examples, along with the .static and .out files. Determine how example output is currently compared, then verify that dynamic fields such as latest height, block hash, and epoch can be excluded while the remaining output is still checked and matching examples do not regenerate .out files.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.