diffplug / diffplug/selfie

[jvm] Snapshots are not being written to disk when there are tests with numbered names

Open
#540 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Kotlin
Stars
101
Forks
18
PR merge metrics
No merged PRs in 30d

Description

There is an issue when creating test methods that are numbered (e.g. `test1()`, `test2()`, ...). At some point snapshots of tests are no longer written to disk. I discovered this while writing integration tests for a Spring Boot app that I generically name `test1()`, `test2()` etc. because I use `@DisplayName` anyway.

I cannot disclose the actual code but here is an example test file using JUnit 5, Java 21 and Selfie `2.5.3` that I could reproduce this with:

```java
import com.diffplug.selfie.Selfie;
import org.junit.jupiter.api.Test;

class ReproduceSelfieBugTest {
//SELFIEWRITE
@Test
void test() {
Selfie.expectSelfie("0").toMatchDisk();
}

@Test
void test1() {
Selfie.expectSelfie("1").toMatchDisk();
}

@Test
void test2() {
Selfie.expectSelfie("2").toMatchDisk();
}

@Test
void test3() {
Selfie.expectSelfie("3").toMatchDisk();
}

@Test
void test4() {
Selfie.expectSelfie("4").toMatchDisk();
}

@Test
void test5() {
Selfie.expectSelfie("5").toMatchDisk();
}

@Test
void test6() {
Selfie.expectSelfie("6").toMatchDisk();
}

@Test
void test7() {
Selfie.expectSelfie("7").toMatchDisk();
}

@Test
void test8() {
Selfie.expectSelfie("8").toMatchDisk();
}

@Test
void test9() {
Selfie.expectSelfie("9").toMatchDisk();
}

@Test
void test10() {
Selfie.expectSelfie("10").toMatchDisk();
}

@Test
void testFoo() {
Selfie.expectSelfie("testFoo").toMatchDisk();
}

@Test
void testFoo1() {
Selfie.expectSelfie("testFoo1").toMatchDisk();
}

@Test
void testFoo2() {
Selfie.expectSelfie("testFoo2").toMatchDisk();
}

@Test
void testFoo3() {
Selfie.expectSelfie("testFoo3").toMatchDisk();
}

@Test
void testFoo4() {
Selfie.expectSelfie("testFoo4").toMatchDisk();
}

@Test
void testFoo5() {
Selfie.expectSelfie("testFoo5").toMatchDisk();
}

@Test
void testFoo6() {
Selfie.expectSelfie("testFoo6").toMatchDisk();
}

@Test
void testFoo7() {
Selfie.expectSelfie("testFoo7").toMatchDisk();
}

@Test
void testFoo8() {
Selfie.expectSelfie("testFoo8").toMatchDisk();
}

@Test
void testFoo9() {
Selfie.expectSelfie("testFoo9").toMatchDisk();
}

@Test
void testFoo10() {
Selfie.expectSelfie("testFoo10").toMatchDisk();
}

@Test
void foo() {
Selfie.expectSelfie("foo").toMatchDisk();
}

@Test
void foo1() {
Selfie.expectSelfie("foo1").toMatchDisk();
}

@Test
void foo2() {
Selfie.expectSelfie("foo2").toMatchDisk();
}

@Test
void foo3() {
Selfie.expectSelfie("foo3").toMatchDisk();
}

@Test
void foo4() {
Selfie.expectSelfie("foo4").toMatchDisk();
}

@Test
void foo5() {
Selfie.expectSelfie("foo5").toMatchDisk();
}

@Test
void foo6() {
Selfie.expectSelfie("foo6").toMatchDisk();
}

@Test
void foo7() {
Selfie.expectSelfie("foo7").toMatchDisk();
}

@Test
void foo8() {
Selfie.expectSelfie("foo8").toMatchDisk();
}

@Test
void foo9() {
Selfie.expectSelfie("foo9").toMatchDisk();
}

@Test
void foo10() {
Selfie.expectSelfie("foo10").toMatchDisk();
}
}
```

Running this test file in IntelliJ 2024.3 produces this `ReproduceSelfieBugTest.ss` file:
```ss
╔═ foo ═╗
foo
╔═ foo1 ═╗
foo1
╔═ foo2 ═╗
foo2
╔═ foo3 ═╗
foo3
╔═ foo4 ═╗
foo4
╔═ foo5 ═╗
foo5
╔═ foo6 ═╗
foo6
╔═ foo7 ═╗
foo7
╔═ foo8 ═╗
foo8
╔═ foo9 ═╗
foo9
╔═ test ═╗
0
╔═ test1 ═╗
1
╔═ test2 ═╗
2
╔═ test3 ═╗
3
╔═ test4 ═╗
4
╔═ test5 ═╗
5
╔═ test6 ═╗
6
╔═ test7 ═╗
7
╔═ test8 ═╗
8
╔═ test9 ═╗
9
╔═ testFoo ═╗
testFoo
╔═ testFoo1 ═╗
testFoo1
╔═ testFoo2 ═╗
testFoo2
╔═ testFoo3 ═╗
testFoo3
╔═ testFoo4 ═╗
testFoo4
╔═ testFoo5 ═╗
testFoo5
╔═ testFoo6 ═╗
testFoo6
╔═ testFoo7 ═╗
testFoo7
╔═ testFoo8 ═╗
testFoo8
╔═ testFoo9 ═╗
testFoo9
╔═ [end of file] ═╗
```

You will notice the snapshots for these tests are missing: `foo10()`, `test10()`, `testFoo10`, i.e. all tests ending with `10`.

Additional information:
If you remove ALL tests except `test()`, `test1()` and `test10()` then the snapshot for `test10()` is _correctly_ saved to disk.
After that, if `test2()` is added back and after re-running the test file, the snapshot for `test10()` is **missing again**. Then, after removing `test2()` once more and re-running the test file, the snapshot for `test10()` is **STILL** missing until the test file is run **one more time**.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.