duckdb / duckdb/duckdb

Unexpected behavior/Error when comparing VARCHAR to JSON-extracted values via ->

Open
#20,596 0 comments 2 reactions 0 assignees View on GitHub
reproduced
Dominant language
C++
Stars
41.2k
Forks
3.8k
Avg merge
2d 9h
Merged PRs (30d)
496

Description

### What happens?

When comparing JSON-extracted values using the `->` operator, DuckDB performs implicit value coercion for numeric and date types, but not for VARCHAR.

JSON strings such as "1" and "2025-01-01" compare equal to INTEGER and DATE literals without explicit casting. In contrast, comparing a VARCHAR to a JSON string requires an explicit cast to avoid a conversion error, and even then the comparison evaluates to false because the JSON string’s quotes are preserved.

This behavior is inconsistent with other type comparisons. Casting a JSON string to VARCHAR should yield the underlying unquoted string value, or string comparisons should apply the same value-level coercion that is already performed for numeric and date types. Preserving JSON string quotes in VARCHAR casts leads to inconsistent and surprising comparison behavior.

For example:
```sql
CREATE OR REPLACE TABLE jason (j JSON);
INSERT INTO jason VALUES
('{ "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] }'),;

SELECT
1 = (jason.j -> '$.id') as int_comparison,
'2025-01-01'::date = (jason.j -> '$.created_at') as date_comparison,
'anatidae' = (jason.j -> '$.family')::varchar as varchar_comparison, --NOTE this is the only one that needs to be cast to prevent an error
'anatidae' = TRIM(jason.j -> '$.family', '"') as trim_comparison
FROM jason;
```
```
┌────────────────┬─────────────────┬────────────────────┬─────────────────┐
│ int_comparison │ date_comparison │ varchar_comparison │ trim_comparison │
│ boolean │ boolean │ boolean │ boolean │
├────────────────┼─────────────────┼────────────────────┼─────────────────┤
│ true │ true │ false │ true │
└────────────────┴─────────────────┴────────────────────┴─────────────────┘
```

### To Reproduce

```
CREATE OR REPLACE TABLE jason (j JSON);
INSERT INTO jason VALUES
('{ "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] }'),;

CREATE OR REPLACE TABLE farchar (i INT, d DATE, v VARCHAR);
INSERT INTO farchar VALUES
(1, '2025-01-01'::date, 'anatidae'),;
```

This first case fails as expected because we compare VARCHAR to a JSON, but for some reason comparisons against other types do not fail (see below)
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = (jason.j -> '$.family');
Conversion Error:
Malformed JSON at byte 0 of input: unexpected character. Input: "anatidae" when casting from source column v

LINE 4: ON farchar.v = (jason.j -> '$.family');
```

This query succeeds but returns unexpected results because double quotes are preserved in the cast to VARCHAR
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = (jason.j -> '$.family')::VARCHAR;
┌──────────┬──────┐
│ v │ j │
│ varchar │ json │
├──────────┼──────┤
│ anatidae │ NULL │
└──────────┴──────┘
```

Here is the workaround, either works.
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = TRIM((jason.j -> '$.family'), '"');
┌──────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ v │ j │
│ varchar │ json │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ anatidae │ { "created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└──────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = TRIM((jason.j -> '$.family')::VARCHAR, '"');
┌──────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ v │ j │
│ varchar │ json │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ anatidae │ { "created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└──────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
```

## Other types work!
Now interestingly comparisons against anything else other than VARCHARs will work...
A date:
```
D SELECT
*
FROM farchar
LEFT JOIN jason
ON farchar.d = (jason.j -> '$.created_at');
┌───────┬────────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ i │ d │ v │ j │
│ int32 │ date │ varchar │ json │
├───────┼────────────┼──────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ 2025-01-01 │ anatidae │ { "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└───────┴────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```

An int:
```
D SELECT
*
FROM farchar
LEFT JOIN jason
ON farchar.i = (jason.j -> '$.id');
┌───────┬────────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ i │ d │ v │ j │
│ int32 │ date │ varchar │ json │
├───────┼────────────┼──────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ 2025-01-01 │ anatidae │ { "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└───────┴────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```

### OS:

MacOS

### DuckDB Version:

1.4.2

### DuckDB Client:

Python

### Hardware:

_No response_

### Full Name:

Kyle Cheung

### Affiliation:

Greybeam

### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

- [x] Yes, I have

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant data sets for reproducing the issue?

Yes

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.