oxidecomputer / oxidecomputer/omicron
basic tests could be much better automated
@davepacheco is already working on this.
Since Mar 17, 2022.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Today, every major resource in the public API has its own hand-written tests that test mostly the same things around basic CRUD. There are also many edge cases not tested in most places (e.g., attempting to rename something to the name of something else that already exists).
It would be neat if instead, we had some configuration data that defines for each resource:
- its URL
- which parts of CRUD it supports
- if "create" is supported, a sample input to create one (maybe where the "name" can be varied somehow)
- if "update" is supported, a sample input to update one (maybe where the "name" can be varied somehow)
This is pretty much the metadata that currently drives the "unauthorized.rs" test.
With this metadata, we could have a test that enumerates all the resources and runs through a battery of CRUD tests:
- basic CRUD:
- list collection while empty (expect 200 with no items)
- PUT/GET/DELETE a non-existent item (expect 404)
- some other random HTTP method on a non-existent item (expect 405)
- create an item (expect 201)
- list again, get again, PUT again (expect successes, check output)
- some other random HTTP method (expect 405)
- delete again (expect success)
- PUT/GET/DELETE again (expect 404)
- list again (check output)
- pagination
- create 30 of them
- paginate through the list with different offsets and check for expected output
- optional / configurable
- scale test: create 10,000 of them?
- more cases and error handling
- invalid input to "create" and "update"
- unexpected body on "get" or "delete"
- successful rename
- attempt to create a new item with a conflicting name
- attempt to rename an existing item to one with an existing name
- successful move of item to another collection (I'm not sure this exists in the API today)
- move item in a way that conflicts with another item's name? (same)
- if it's a collection: attempt to remove the collection while it's non-empty
Much of this behavior is implemented in common code, but it still has to be applied correctly to every endpoint.
A system like this would probably be a lot less code than we have today while covering a lot more cases and ensuring consistency of behavior across endpoints.
We could use this to drive a CLI like oxapi_demo (i.e., CRUD for every resource, with sample input data).
We may be able to start with (or even just use) the metadata in the unauthorized.rs test.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.