apple / apple/pkl

Unable to create test for non-empty constraint on listing

Open
#461 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
11.5k
Forks
402
Avg merge
1d 15h
Merged PRs (30d)
20

Description

The intention here is to create a test for checking a non-empty constraint on a listing, for regression purposes. I realize that `pkl test` has not been documented, so it's quite possible there are some misunderstandings or unsupported functionality being used here.

We need three files for a demonstration.

```
# NonEmptyDef.pkl:
values: Listing(!isEmpty)

# NonEmptyImpl.pkl:
amends "NonEmptyDef.pkl"

values {
}

# NonEmptyTest.pkl:
amends "pkl:test"
import "pkl:test"
import "NonEmptyImpl.pkl" as confUnderTest

examples {
["basic-test"] {
test.catch(() -> confUnderTest)
}
}
```

We can clearly see that `values` defined in `NonEmptyImpl` is invalid, as the listing is required to not be empty. However, when running this test:
```
$ pkl test --overwrite NonEmptyTest.pkl
module NonEmptyTest (file:///tmp/issues/NonEmptyTest.pkl, line 1)
basic-test ✍️
NonEmptyTest ❌
Error:
-- Pkl Error --
Expected an exception, but none was thrown.

7 | test.catch(() -> confUnderTest)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
at NonEmptyTest#examples["basic-test"][#1] (file:///tmp/issues/NonEmptyTest.pkl, line 7)
```

The error seems to suggest that there is no evaluation error here. If that's the case, we should be able to try the inverse test, as demonstrated in NonEmptyTest-Inv.pkl:
```
amends "pkl:test"
import "pkl:test"
import "NonEmptyImpl.pkl" as confUnderTest

examples {
["basic-test"] {
confUnderTest
}
}
```

This time we get the opposite error instead; an error was thrown and we did not catch it:
```
$ pkl test --overwrite NonEmptyTest-Inv.pkl
module NonEmptyTest-Inv (file:///tmp/issues/NonEmptyTest-Inv.pkl, line 1)
basic-test ✍️
NonEmptyTest-Inv ❌
Error:
-- Pkl Error --
Type constraint `!isEmpty` violated.
Value: new Listing {}

1 | values: Listing(!isEmpty)
^^^^^^^^
at NonEmptyDef#values (file:///tmp/issues/NonEmptyDef.pkl, line 1)

3 | values {
^^^^^^^^
at NonEmptyImpl#values (file:///tmp/issues/NonEmptyImpl.pkl, line 3)
```

If we instead embed this list inside a mapping, this works as expected:
```
# NonEmptyDef2.pkl:
stuff: Mapping(!isEmpty)>

# NonEmptyImpl2.pkl:
amends "NonEmptyDef2.pkl"

stuff {
["key"] {
}
}

# NonEmptyTest2.pkl:
amends "pkl:test"
import "pkl:test"
import "NonEmptyTest2.pkl" as confUnderTest

examples {
["basic-test"] {
test.catch(() -> confUnderTest.stuff)
}
}
```
```
$ pkl test --overwrite NonEmptyTest2.pkl
module NonEmptyTest2 (file:///tmp/issues/NonEmptyTest2.pkl, line 1)
basic-test ✍️

$ cat NonEmptyTest2.pkl-expected.pcf
examples {
["basic-test"] {
"Type constraint `!isEmpty` violated. Value: new Listing {}"
}
}
```

The above has been tested using the following versions:
- `Pkl 0.25.3 (Linux 5.15.0-1053-aws, native)`
- `Pkl 0.26.0-dev+3a31188 (Linux 5.4.0-177-generic, native)`

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the three-file NonEmptyDef.pkl, NonEmptyImpl.pkl, and NonEmptyTest.pkl examples with `pkl test --overwrite`, then compare them with the mapping example in NonEmptyDef2.pkl and NonEmptyTest2.pkl. Investigate why direct listing evaluation differs from evaluation inside a mapping, and verify that the expected exception is caught and recorded in the generated .pcf output.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.