dbt-labs / dbt-labs/spark-utils

Add compatibility for rest of dbt-utils

Open
#7 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36
Forks
14
PR merge metrics
No merged PRs in 30d

Description

Macros that need reimplementing:
- [x] `get_url_parameter`: This is actually a fix for `split_part`, which should only add escape characters (`\\`) if the delimiter needs escaping. Fix: #8
- [ ] `get_relations_by_prefix_sql`, `get_tables_by_pattern_sql`, `get_tables_by_prefix`, `get_tables_by_pattern`: Need rethinking. Spark doesn't have a way of querying an `information_schema`, only `show`/`describe` commands—which do have access to regex-y `like` but cannot be chained/unioned together. Fix: #10

Integration tests currently failing for macros that otherwise work:
- [ ] `test_groupby`: `Star (*) is not allowed in select list when GROUP BY ordinal position is used;`
- [ ] `test_recency`: This is a product of how the integration test is currently written, will be fixed by https://github.com/fishtown-analytics/dbt-utils/pull/318
- [ ] `test_get_relations_by_pattern`: Requires support for schema patterns, which `spark_utils.get_relations_by_pattern` doesn't (yet) offer
- [x] `test_equal_column_subset`: A minor bug with `dbt_utils.equality` when used with an ephemeral model + an explicit column subset. Fix: https://github.com/fishtown-analytics/dbt-utils/pull/321
- [x] `assert_get_query_results_as_dict_objects_equal`: The query is not ordered. The way the test is written, if the order in `expected_dictionary` differs from `actual_dictionary`, the test fails. Fix: https://github.com/fishtown-analytics/dbt-utils/pull/322

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the listed unresolved macros and the failing integration tests, especially get_relations_by_prefix_sql, get_tables_by_pattern, test_groupby, test_recency, and test_get_relations_by_pattern. Run the integration tests to reproduce the failures; done means the remaining compatibility items are implemented and the affected tests pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.