corvus-dotnet / corvus-dotnet/Corvus.Python

Add Gherkin-based testing for Power BI semantic models

Open
#64 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

Add a BDD testing capability to the package so that analysts can specify expected
behaviour of Power BI semantic models in Gherkin, and have those specifications executed
against the live models from a Fabric notebook.

All logic lives in the package. The notebook is a wrapper, and the `.feature` files live in
the notebook's built-in Resources folder so they are editable in Fabric and versioned by
workspace Git integration.

## Motivation

Semantic model logic is currently untested. Measure changes, model refactors and RLS
changes are validated by eye, and regressions surface in reports rather than in a build.
Existing BDD approaches for Power BI depend on .NET tooling or the ExecuteQueries REST API
and do not fit a Fabric-native workflow.

Semantic Link (`sempy.fabric.evaluate_dax`) removes the blocker: DAX can be executed from a
notebook with no XMLA client libraries, no connection strings and no service principal,
authenticating as the executing identity.

Writing the specifications in Gherkin means the people who know what the numbers should be
can author and review the tests without writing Python.

## Scope

Target notebook body, in full:

```python
from corvus.python.testing import run_tests

result = run_tests(features="builtin/features", tags=tags, workspace=workspace)
result.display()
result.raise_if_failed()
```

**In scope**

- [ ] `corvus.python.testing` subpackage: behave orchestrated in-process, results adapter, public API
- [ ] Reusable step library covering model selection, filter context, measure evaluation
(scalar and grouped), raw DAX, scalar and tabular assertions, model contract
assertions, and RLS via impersonation
- [ ] HTML report rendered inline in the notebook
- [ ] `validate_features()` dry run — parses features and resolves steps without querying
a model, so it can gate PRs in CI with no Fabric capacity
- [ ] Optional persistence of results to a Delta table, and JUnit XML output
- [ ] Example feature files and wrapper notebook shipped as package data
- [ ] Docs: step vocabulary reference, extension guide, deployment notes
**Out of scope**

- Report-layer (visual) testing
- Data quality testing of the underlying lakehouse tables
- Authoring UI for feature files beyond what Fabric and VS Code already provide
## Design decisions already taken

- **behave, not a bespoke runner.** The package already depends on behave. A prototype
with a hand-rolled parser exists and its parser, step registry, context and run loop are
all being discarded — see `IMPLEMENTATION-PLAN.md` §2 for the keep/discard table.
- **Feature files in notebook Resources, not a lakehouse or a separate repo.** Resources
are committed to Git with the notebook, which is the only option satisfying both
"editable in Fabric" and "under source control". Lakehouse file content is not Git-synced.
- **behave runs against a generated temp directory.** behave requires a `steps/` directory
adjacent to the features. Generating it into a throwaway copy keeps the Resources folder
to `.feature` files only, so analysts never see package plumbing and it never appears in
a PR diff.
- **A pluggable query engine.** `set_query_engine()` allows the whole suite to run against
a fake in CI, with no Fabric capacity and no network.
- **The package must import without a Fabric runtime.** sempy is imported lazily.
## Tasks

1. [ ] Spike behave in-process: temp run directory, hook injection, results adapter,
one trivial feature. Resolves most of the open questions below.
2. [ ] Port `engine.py` (DAX execution, engine override, cache) and `_compare.py`
(column and value normalisation, frame comparison) with unit tests
3. [ ] Port the step library to behave decorators and `parse` patterns, module by module
4. [ ] Port the HTML reporter
5. [ ] Add `validate_features`, Delta persistence and JUnit passthrough
6. [ ] Wrapper notebook, packaged examples, README
7. [ ] End-to-end verification against a real semantic model in a dev workspace
## Open questions to resolve during implementation

- [ ] Does assigning to `runner.hooks[...]` survive `Runner.run()`, or is a generated
`environment.py` required?
- [ ] Is a `null` formatter registered in our behave version? What is the quietest config?
- [ ] Does `Configuration(command_args=..., load_config=False)` reliably ignore `sys.argv`
and local ini files inside a Fabric notebook?
- [ ] Which `Status` enum members exist in our behave version?
- [ ] Does scenario-scoped context state leak between scenarios?
- [ ] `INFO.MEASURES()` availability on our models, vs the `$SYSTEM.TMSCHEMA_*` fallback
- [ ] `sempy_labs.evaluate_dax_impersonation` argument names, if RLS steps are in scope
- [ ] Resources-in-Git limits and behaviour in our tenant — it is recent, opt-in per
notebook, and has file size and count caps
## Risks

- **Precision handling.** Assertions compare to the precision written in the feature file,
so `12,477,304.29` passes for `12477304.2871`. This depends on the `parse` type converter
retaining the literal text rather than returning only a float. A careless port silently
turns these into exact matches and produces confusing failures on any measure with
floating-point tails.
- **Step text collisions.** behave raises `AmbiguousStep` at import time, affecting every
consumer of the package. Check existing step definitions before adding.
- **Environment startup cost.** Attaching a custom Spark Environment adds noticeable
session startup time, and republishing after a library change takes minutes. Fine for a
nightly suite; if a post-refresh smoke test needs to run in seconds, the fallback is a
Python notebook installing from the private feed.
- **Query latency.** Each distinct DAX query is a round trip. The prototype caches
identical queries within a run so a shared `Background` costs one query, not one per
scenario. Preserve this.
## Acceptance criteria

- [ ] The notebook contains only an import, a `run_tests` call, `display()` and
`raise_if_failed()`
- [ ] The Resources folder contains `.feature` files and nothing else
- [ ] `pytest` passes with no Fabric capacity and no network
- [ ] A failing scenario renders the assertion message, the expected table, the actual rows
and the generated DAX
- [ ] `validate_features` catches an undefined step without executing a query
- [ ] behave tag expressions (`"@smoke and not @slow"`) work
- [ ] An analyst can add a scenario, open a PR and have it reviewed without touching Python

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.