dbt-labs / dbt-labs/dbt-adapters

[Bug] Index errors when using split_part

Open
#469 0 comments 0 reactions 0 assignees View on GitHub
pkg:dbt-spark type:bug
Dominant language
Python
Stars
233
Forks
362
Avg merge
3d 22h
Merged PRs (30d)
9

Description

### Is this a new bug in dbt-spark?

- [X] I believe this is a new bug in dbt-spark
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

When called with a part index that is out of bounds, and ansi-mode on, the split_part macro leads to an exception

### Expected Behavior

Per the tests in BaseSplitPart in the adapter tests, the expectation is that this macro can be invoked with part indexes greater than the number of parts generated without throwing an exception specifically this row in the seed:
```
,|,,,,
```

We can accommodate this behavior by using get, rather than indexing the array, but only in Spark 3.4.0 or later.

### Steps To Reproduce

1. set spark.sql.ansi.enabled=true
2. use split_part passing an out of bounds index
3. observe exception

### Relevant log output

_No response_

### Environment

This issue has been in there for a while, but I'm just hitting it now due to new defaults in a Databricks environment I was asked to test against.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.