[CT-3311] Handle various edge cases with partially parsing yaml-only nodes
- Dominant language
- Rust
- Stars
- 13.8k
- Forks
- 2.6k
- Avg merge
- 21h 31m
- Merged PRs (30d)
- 56
Description
### Description
When partially parsing schema file only nodes, calls to "schedule_nodes_for_parsing" will not always correctly handle additional schema file changes associated with nodes that are invalidated by the current schema file change. In some cases it will pull in the schema file "element" from the saved manifest, instead of the current updated manifest, thereby perpetuating the old state of the element. When the partial parser was originally written this problem was avoided by handling schema file changes first, so that SQL file changes were already referencing the correct schema file elements. But since then we have added new node types that are entirely schema file based and have dependent nodes which must be scheduled for reparsing, and the partial parser does not handle that well.
This was partially handled by a previous pull request which added the "new_patch" argument to the "merge_patch" method, but there are additional cases which might also cause problems, such as when an element is deleted which was also referenced by a dependent node which is scheduled for parsing.
In addition partial parsing does not currently handle yaml only elements with duplicate names (such as exposures) and a solution to the other problem could also probably handle that issue.
One way of handling this issue would be to introduce another stage into partial parsing and "pre-parse" schema files to create file associated dictionaries of element name to element "block", prior to starting the actual reparse scheduling (in handle_schema_files). This would require switching from getting the yaml patch from the "saved_manifest" to getting the yaml patches from the pre-parsed blocks.
### Acceptance Criteria
Partial parsing handles deletion of yaml elements from nodes that are also associated with changed yaml-only node types (such as exposures and groups). Partial parsing handles detection of duplicate element blocks.
### Impact to other teams
None
### Will backports be required?
Maybe, at product management discretion. These errors tend to be edge cases and require very specific conditions.
### Context
Contributor guide
Assessment
This issue has not been assessed yet.