IntersectMBO / IntersectMBO/cardano-ledger
Byron era test failure on CI
- Dominant language
- Haskell
- Stars
- 295
- Forks
- 179
- Avg merge
- 4d 7h
- Merged PRs (30d)
- 29
Description
This failure can be reproduced at this commit: 1f0c6a4eaa4fb8f937c30a4608a4fafedaca216e
It is not really a failure of test, instead it is failure of the requirement on the data generated. Considering this is Byron era, it would be sensible to simply relax this requirement, especially since this failure is extremely rare.
```
We are generating reasonable Chain Traces: FAIL (2.76s)
✗ We are generating reasonable Chain Traces failed
after 200 tests.
shrink path: 200:
there are more certificates than blocks 7% █▍·················· ✓ 1%
at most 75% of blocks have no certificates 21% ████▏··············· ✓ 10%
at least 25% of delegates will delegate to this epoch 34% ██████▉············· ✓ 25%
at least 50% of delegations will delegate to the next epoch 86% █████████████████▏·· ✓ 60%
at most 30% of certificates will self-delegate 24% ████▋··············· ✓ 10%
at least 25% delegates have multiple delegators 77% ███████████████▍···· ✓ 60%
some delegates have at least 5 corresponding delegators 36% ███████▎············ ✓ 20%
at most 50% of delegators change their delegation 16% ███················· ✓ 5%
some delegators have changed their delegation 5 or more times 36% ███████▎············ ✓ 20%
⚠ at most 25% of delegations are repeats 1% ▏··················· ✗ 2%
some delegations are repeated 10 or more times 42% ████████▍··········· ✓ 30%
some blocks have 5 or more certificates 30% ██████·············· ✓ 15%
there is at least one change of epoch in the trace 78% ███████████████▋···· ✓ 50%
there are at least 5 epoch changes in the trace 42% ████████▍··········· ✓ 30%
┏━━ test/Test/Byron/Spec/Chain/STS/Properties.hs ━━━
110 ┃ relevantCasesAreCovered :: Property
111 ┃ relevantCasesAreCovered = withTests 200 $
112 ┃ property $ do
113 ┃ tr <- forAll $ traceSigGen () (Desired 200) (sigGenChain GenDelegation NoGenUTxO NoGenUpdate)
114 ┃ let certs = traceDCerts tr
115 ┃
116 ┃ -- for at least 1% of traces...
117 ┃ cover
118 ┃ 1
119 ┃ "there are more certificates than blocks"
170 ┃ -- for at least 2% of traces...
171 ┃ cover
┃ ^^^^^
┃ │ Failed (1% coverage)
172 ┃ 2
173 ┃ "at most 25% of delegations are repeats"
174 ┃ (repeatedDelegationsRatio certs <= 0.25)
175 ┃
176 ┃ -- for at least 30% of traces...
177 ┃ cover
178 ┃ 30
179 ┃ "some delegations are repeated 10 or more times"
180 ┃ (10 <= maxRepeatedDelegations certs)
181 ┃
182 ┃ -- for at least 15% of traces...
183 ┃ cover
184 ┃ 15
185 ┃ "some blocks have 5 or more certificates"
186 ┃ (5 <= maxCertsPerBlock (traceDCertsByBlock tr))
187 ┃
188 ┃ -- for at least 50% of traces...
189 ┃ cover
190 ┃ 50
191 ┃ "there is at least one change of epoch in the trace"
192 ┃ (2 <= epochBoundariesInTrace tr)
193 ┃
194 ┃ -- for at least 30% of traces...
195 ┃ cover
196 ┃ 30
197 ┃ "there are at least 5 epoch changes in the trace"
198 ┃ (5 <= epochBoundariesInTrace tr)
199 ┃ where
200 ┃ -- Get the epoch in which the delegation certificates of the trace were
201 ┃ -- applied, paired with the epoch of the delegation certificate.
202 ┃ epochDelegationEpoch :: Trace CHAIN -> [(Epoch, Epoch)]
203 ┃ epochDelegationEpoch tr =
204 ┃ preStatesAndSignals @CHAIN OldestFirst tr
205 ┃ & fmap (sEpoch_ . view _1 *** (fmap depoch . (_bDCerts . _bBody)))
206 ┃ & fmap (\(e, es) -> zip (repeat e) es)
207 ┃ & concat
208 ┃ where
209 ┃ blockCount = _traceEnv tr ^. _5
210 ┃ sEpoch_ = flip sEpoch blockCount
211 ┃
212 ┃ -- Count the number of epoch boundaries in the trace
213 ┃ epochBoundariesInTrace :: Trace CHAIN -> Int
214 ┃ epochBoundariesInTrace tr =
215 ┃ length $
216 ┃ filter (== 0) (isAtBoundary <$> slots)
217 ┃ where
218 ┃ blocks = traceSignals NewestFirst tr
219 ┃ slots = blocks ^.. traverse . bHeader . bhSlot
220 ┃ k = _traceEnv tr ^. _5
221 ┃ isAtBoundary = (`rem` slotsPerEpoch k) . unSlot
Labels not sufficiently covered after 200 tests
Use "--pattern '$NF ~ /We are generating reasonable Chain Traces/' --hedgehog-replay '200: Seed 1712611033913888896 11884840232036325667'" to reproduce from the command-line.
Use -p '/We are generating reasonable Chain Traces/' to rerun this test only.
```
Contributor guide
Research direction
Start by reproducing the failure with the provided commit and Hedgehog replay command, then inspect test/Test/Byron/Spec/Chain/STS/Properties.hs around relevantCasesAreCovered and the coverage check at lines 170-174. Adjust the Byron-era coverage requirement as requested and rerun the focused test until the coverage labels pass reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- haskell
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100