Tests used in CI should be defined in a single table called `dolt_tests`
- Dominant language
- Go
- Stars
- 24.4k
- Forks
- 873
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 121
Description
The interface to integration tests that currently exists is:
1. Define a saved query for your test
2. Define a workflow.yaml file with the saved query and it's assertions.
3. Upload the workflow.yaml file using `dolt ci init` and `dolt ci upload`
This made sense when testing was going to be a DoltHub specific thing. But now that testing is required for agentic use cases, this interface is cumbersome.
My proposal is to separate the interface into two. We have a system table called `dolt_tests` to define tests and assertions. We keep the YAML file interface to have DoltHub run tests on push or on Pull Request. Thus, testing becomes a Dolt feature and CI becomes a DoltHub (or DoltLab or Dolt Workbench) feature.
I propose the schema for `dolt_tests` be:
`test_group varchar(100)`
`test_name varchar(100)`
`test_query varchar(1000)`
`test_assertion varchar(100)`
You then can run the tests with `dolt test run ` on the command line or `call dolt_test_run('')` in SQL. Omission of test_group runs all tests.
This has the advantage of defining queries and assertions in the same place. It is also more obvious to the user.
I think in the YAML interface stays but leverages the fact that dolt test exists. This:
```yaml
name: grade distribution validation
on:
push:
branches:
- main
- master
- test-violations
- update-ci-branches
- passing-grades-expansion
jobs:
- name: schema validation
steps:
- name: verify all required tables exist
saved_query_name: table_existence_check
expected_rows: "== 1"
expected_columns: "== 1"
- name: check referential integrity
saved_query_name: referential_integrity_check
expected_rows: "== 1"
expected_columns: "== 1"
- name: grade distribution compliance
steps:
- name: verify no grade distribution violations
saved_query_name: grade_distribution_violations
expected_rows: "== 0"
```
becomes something like:
```yaml
name: grade distribution validation
on:
push:
branches:
- main
- master
- test-violations
- update-ci-branches
- passing-grades-expansion
jobs:
- name: schema validation
steps:
- dolt_test_run: table_existence_check
- dolt_test_run: referential_integrity_check
- name: grade distribution compliance
steps:
- dolt_test_run: grade_distribution_violations
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.