dbt-labs / dbt-labs/spark-utils
Add compatibility for rest of dbt-utils
- Dominant language
- Python
- Stars
- 36
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
Macros that need reimplementing:
- [x] `get_url_parameter`: This is actually a fix for `split_part`, which should only add escape characters (`\\`) if the delimiter needs escaping. Fix: #8
- [ ] `get_relations_by_prefix_sql`, `get_tables_by_pattern_sql`, `get_tables_by_prefix`, `get_tables_by_pattern`: Need rethinking. Spark doesn't have a way of querying an `information_schema`, only `show`/`describe` commands—which do have access to regex-y `like` but cannot be chained/unioned together. Fix: #10
Integration tests currently failing for macros that otherwise work:
- [ ] `test_groupby`: `Star (*) is not allowed in select list when GROUP BY ordinal position is used;`
- [ ] `test_recency`: This is a product of how the integration test is currently written, will be fixed by https://github.com/fishtown-analytics/dbt-utils/pull/318
- [ ] `test_get_relations_by_pattern`: Requires support for schema patterns, which `spark_utils.get_relations_by_pattern` doesn't (yet) offer
- [x] `test_equal_column_subset`: A minor bug with `dbt_utils.equality` when used with an ephemeral model + an explicit column subset. Fix: https://github.com/fishtown-analytics/dbt-utils/pull/321
- [x] `assert_get_query_results_as_dict_objects_equal`: The query is not ordered. The way the test is written, if the order in `expected_dictionary` differs from `actual_dictionary`, the test fails. Fix: https://github.com/fishtown-analytics/dbt-utils/pull/322
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the listed unresolved macros and the failing integration tests, especially get_relations_by_prefix_sql, get_tables_by_pattern, test_groupby, test_recency, and test_get_relations_by_pattern. Run the integration tests to reproduce the failures; done means the remaining compatibility items are implemented and the affected tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100