dbt-labs / dbt-labs/dbt-common
[Feature] Use fastjsonschema for schema validation to speed up parse times
- Dominant language
- Python
- Stars
- 40
- Forks
- 42
- PR merge metrics
- No merged PRs in 30d
Description
### Is this your first time submitting a feature request?
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [x] I have searched the existing issues, and I could not find an existing issue for this feature
- [x] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion
### Describe the feature
Use `fastjsonschema` for initial json schema validation and then run the current `jsonschema.Draft7Validator.iter_errors` only when there are errors. On my local, `fastjsonschema` was ~5x faster. Since the dominant case is no errors, this results in 5x speedup on average, with very minor slowdown in error cases.
Profiled this on a project with ~19k paths (models, macros, etc.) and saw a 12.4% reduction in parse time. Also tested on smaller projects (~500 and ~1k paths) and did not see any degradation.
Mainly impacts two call sites:
1. **`dbt_common.dataclass_schema.dbtClassMixin.validate()`** (the per-node mashumaro-schema check)
2. **`dbt_core.jsonschemas.jsonschemas._validate_with_schema()`** (the dbt_project.yml / resource-yml deprecation pass)
### Describe alternatives you've considered
- **Replace `jsonschema` with `fastjsonschema` outright**. `fastjsonschema` raises a single `JsonSchemaException` on the first error and does not preserve the rich `jsonschema.ValidationError` shape that `jsonschema_validate()` / `validate_model_config()` rely on for dispatching `additionalProperties` / `anyOf` / `type` deprecation warnings. Keeping the slow path for the failure case is essential.
- **Cache `Draft7Validator` instances per schema**: a `Draft7Validator` is cheap to construct; the cost is in `iter_errors` itself, not in instantiation. Negligible win.
- **Switch to `python-fastjsonschema`'s `compile_to_code` + ship the code**: would require shipping generated `.py` files and re-running compile on every schema bump. The in-memory cache keyed by class is functionally equivalent and has no build-step impact.
- **Pre-validate on yml read instead of per-node during parse**: would require restructuring the parser. Same total work, same total time, harder to roll back if it regresses.
- **Tighten the schemas themselves so `Draft7Validator` runs less**: addresses cause not symptom, but the schemas are auto-generated from mashumaro types and from `dbt-protos`. So trimming them is upstream churn for marginal gain.
### Who will this benefit?
All dbt users, but mostly those with larger projects where dbt commands run slowly due to slow parse.
### Are you interested in contributing this feature?
Yes, just need review of the idea and contributed code.
### Anything else?
[Example implementation here](https://github.com/macks22/dbt-common/pull/2). Quick summary:
1. **Add `fastjsonschema>=2.19.0,<3.0` to `dependencies`** in `dbt_common/pyproject.toml`.
2. **Wrap `dbtClassMixin.validate(cls, data)` with a fast path** in `dbt_common/dataclass_schema.py`.
1. A new `_get_fast_validator(cls, schema)` helper compiles the per-class schema via `fastjsonschema.compile(schema, formats=_NUMERIC_FORMAT_NOOPS, use_default=False)` and caches the result in a module-level dict keyed by `cls`.
2. On valid data, the fast validator returns immediately and we skip `Draft7Validator(json_schema).iter_errors(data)` entirely.
3. On invalid data (or for schemas `fastjsonschema` cannot compile), the fast validator raises `JsonSchemaException` and we fall through to the slow path. So the resulting `ValidationError.create_from(error)` carries the same context, message, and path that callers depend on.
3. Two `fastjsonschema` configuration knobs are required:
1. `use_default=False`: the default behavior mutates the input dict by injecting schema `default` values. dbt schemas declare `"default": null` for several optional fields, which would inject `None` into typed configs and break the downstream slow-path `Draft7Validator` (it rejects `None` for the corresponding typed field).
2. `formats={int32, int64, uint64, float, double}` no-op format checkers: `fastjsonschema` rejects unknown OpenAPI-style numeric format hints at compile time. The actual numeric constraints are already enforced by `"type": "integer" / "number"`. So no validation is lost. The mashumaro-generated schemas don't currently use these hints, but the no-ops are included defensively for future compatibility.
Contributor guide
Assessment
This issue has not been assessed yet.