Unable to create test for non-empty constraint on listing
- Dominant language
- Java
- Stars
- 11.5k
- Forks
- 402
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 20
Description
The intention here is to create a test for checking a non-empty constraint on a listing, for regression purposes. I realize that `pkl test` has not been documented, so it's quite possible there are some misunderstandings or unsupported functionality being used here.
We need three files for a demonstration.
```
# NonEmptyDef.pkl:
values: Listing(!isEmpty)
# NonEmptyImpl.pkl:
amends "NonEmptyDef.pkl"
values {
}
# NonEmptyTest.pkl:
amends "pkl:test"
import "pkl:test"
import "NonEmptyImpl.pkl" as confUnderTest
examples {
["basic-test"] {
test.catch(() -> confUnderTest)
}
}
```
We can clearly see that `values` defined in `NonEmptyImpl` is invalid, as the listing is required to not be empty. However, when running this test:
```
$ pkl test --overwrite NonEmptyTest.pkl
module NonEmptyTest (file:///tmp/issues/NonEmptyTest.pkl, line 1)
basic-test ✍️
NonEmptyTest ❌
Error:
-- Pkl Error --
Expected an exception, but none was thrown.
7 | test.catch(() -> confUnderTest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at NonEmptyTest#examples["basic-test"][#1] (file:///tmp/issues/NonEmptyTest.pkl, line 7)
```
The error seems to suggest that there is no evaluation error here. If that's the case, we should be able to try the inverse test, as demonstrated in NonEmptyTest-Inv.pkl:
```
amends "pkl:test"
import "pkl:test"
import "NonEmptyImpl.pkl" as confUnderTest
examples {
["basic-test"] {
confUnderTest
}
}
```
This time we get the opposite error instead; an error was thrown and we did not catch it:
```
$ pkl test --overwrite NonEmptyTest-Inv.pkl
module NonEmptyTest-Inv (file:///tmp/issues/NonEmptyTest-Inv.pkl, line 1)
basic-test ✍️
NonEmptyTest-Inv ❌
Error:
-- Pkl Error --
Type constraint `!isEmpty` violated.
Value: new Listing {}
1 | values: Listing(!isEmpty)
^^^^^^^^
at NonEmptyDef#values (file:///tmp/issues/NonEmptyDef.pkl, line 1)
3 | values {
^^^^^^^^
at NonEmptyImpl#values (file:///tmp/issues/NonEmptyImpl.pkl, line 3)
```
If we instead embed this list inside a mapping, this works as expected:
```
# NonEmptyDef2.pkl:
stuff: Mapping(!isEmpty)>
# NonEmptyImpl2.pkl:
amends "NonEmptyDef2.pkl"
stuff {
["key"] {
}
}
# NonEmptyTest2.pkl:
amends "pkl:test"
import "pkl:test"
import "NonEmptyTest2.pkl" as confUnderTest
examples {
["basic-test"] {
test.catch(() -> confUnderTest.stuff)
}
}
```
```
$ pkl test --overwrite NonEmptyTest2.pkl
module NonEmptyTest2 (file:///tmp/issues/NonEmptyTest2.pkl, line 1)
basic-test ✍️
$ cat NonEmptyTest2.pkl-expected.pcf
examples {
["basic-test"] {
"Type constraint `!isEmpty` violated. Value: new Listing {}"
}
}
```
The above has been tested using the following versions:
- `Pkl 0.25.3 (Linux 5.15.0-1053-aws, native)`
- `Pkl 0.26.0-dev+3a31188 (Linux 5.4.0-177-generic, native)`
Contributor guide
Research direction
Start by reproducing the three-file NonEmptyDef.pkl, NonEmptyImpl.pkl, and NonEmptyTest.pkl examples with `pkl test --overwrite`, then compare them with the mapping example in NonEmptyDef2.pkl and NonEmptyTest2.pkl. Investigate why direct listing evaluation differs from evaluation inside a mapping, and verify that the expected exception is caught and recorded in the generated .pcf output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100