Improving the validator

One of the discussions at the workshop was about improving the server validator, given that it currently just checks that endpoints exist and return valid responses, and that filters do not return errors provided they are valid against the specification. We also need to make the validator much more flexible and modular.

One suggestion to really validate an implementation is to run the validator in an exploratory way, by:

• running a general query on an endpoint, e.g. ?filter=nsites<n, increasing n until there are at least some results.
• from the results of the initial query, first check that they all actually obey the filter provided, and then look at the differences between entries and try to construct other queries/conditions on the filter that should return a set of different results. e.g. ?filter=nsites<n AND elements HAS <element found in initial query>.

Using this approach, several features of the filter language can be tested, entries themselves can be validated, and the entries that are returned can be manually verified against the constructed filters.

This approach was partially implemented in #417. Still the following issues remain:

• Query tests on dictionary and timestamp field types (we don't actually have timestamps implemented anyway)
• Query combination with AND and OR
• Query negation with NOT
• Queries with optional operators like HAS ONLY
• Queries on relationships
• Queries with KNOWN and UNKNOWN
• Queries that test operator precedence...
• Queries that test query parameters, e.g. api_hint and sorting
• JSON responses to be used in the providers dashboard
• Test that unknown fields return a 501

Start by reading the partial exploratory validator implementation referenced in #417 and compare it with the remaining checklist in this issue. The work is done when the validator covers the listed query types, combinations, operators, relationships, precedence, and query parameters, while preserving the completed JSON-response and unknown-field checks.