dolthub / dolthub/dolt

Tests used in CI should be defined in a single table called `dolt_tests`

Open
#9,659 9 comments 0 reactions 0 assignees View on GitHub
configuration enhancement system tables version control
Dominant language
Go
Stars
24.4k
Forks
873
Avg merge
1d 9h
Merged PRs (30d)
121

Description

The interface to integration tests that currently exists is:

1. Define a saved query for your test
2. Define a workflow.yaml file with the saved query and it's assertions.
3. Upload the workflow.yaml file using `dolt ci init` and `dolt ci upload`

This made sense when testing was going to be a DoltHub specific thing. But now that testing is required for agentic use cases, this interface is cumbersome.

My proposal is to separate the interface into two. We have a system table called `dolt_tests` to define tests and assertions. We keep the YAML file interface to have DoltHub run tests on push or on Pull Request. Thus, testing becomes a Dolt feature and CI becomes a DoltHub (or DoltLab or Dolt Workbench) feature.

I propose the schema for `dolt_tests` be:

`test_group varchar(100)`
`test_name varchar(100)`
`test_query varchar(1000)`
`test_assertion varchar(100)`

You then can run the tests with `dolt test run ` on the command line or `call dolt_test_run('')` in SQL. Omission of test_group runs all tests.

This has the advantage of defining queries and assertions in the same place. It is also more obvious to the user.

I think in the YAML interface stays but leverages the fact that dolt test exists. This:

```yaml
name: grade distribution validation
on:
push:
branches:
- main
- master
- test-violations
- update-ci-branches
- passing-grades-expansion
jobs:
- name: schema validation
steps:
- name: verify all required tables exist
saved_query_name: table_existence_check
expected_rows: "== 1"
expected_columns: "== 1"

- name: check referential integrity
saved_query_name: referential_integrity_check
expected_rows: "== 1"
expected_columns: "== 1"

- name: grade distribution compliance
steps:
- name: verify no grade distribution violations
saved_query_name: grade_distribution_violations
expected_rows: "== 0"
```

becomes something like:

```yaml
name: grade distribution validation
on:
push:
branches:
- main
- master
- test-violations
- update-ci-branches
- passing-grades-expansion
jobs:
- name: schema validation
steps:
- dolt_test_run: table_existence_check
- dolt_test_run: referential_integrity_check

- name: grade distribution compliance
steps:
- dolt_test_run: grade_distribution_violations
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.