Unexpected behavior/Error when comparing VARCHAR to JSON-extracted values via ->
- Dominant language
- C++
- Stars
- 41.2k
- Forks
- 3.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 496
Description
### What happens?
When comparing JSON-extracted values using the `->` operator, DuckDB performs implicit value coercion for numeric and date types, but not for VARCHAR.
JSON strings such as "1" and "2025-01-01" compare equal to INTEGER and DATE literals without explicit casting. In contrast, comparing a VARCHAR to a JSON string requires an explicit cast to avoid a conversion error, and even then the comparison evaluates to false because the JSON string’s quotes are preserved.
This behavior is inconsistent with other type comparisons. Casting a JSON string to VARCHAR should yield the underlying unquoted string value, or string comparisons should apply the same value-level coercion that is already performed for numeric and date types. Preserving JSON string quotes in VARCHAR casts leads to inconsistent and surprising comparison behavior.
For example:
```sql
CREATE OR REPLACE TABLE jason (j JSON);
INSERT INTO jason VALUES
('{ "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] }'),;
SELECT
1 = (jason.j -> '$.id') as int_comparison,
'2025-01-01'::date = (jason.j -> '$.created_at') as date_comparison,
'anatidae' = (jason.j -> '$.family')::varchar as varchar_comparison, --NOTE this is the only one that needs to be cast to prevent an error
'anatidae' = TRIM(jason.j -> '$.family', '"') as trim_comparison
FROM jason;
```
```
┌────────────────┬─────────────────┬────────────────────┬─────────────────┐
│ int_comparison │ date_comparison │ varchar_comparison │ trim_comparison │
│ boolean │ boolean │ boolean │ boolean │
├────────────────┼─────────────────┼────────────────────┼─────────────────┤
│ true │ true │ false │ true │
└────────────────┴─────────────────┴────────────────────┴─────────────────┘
```
### To Reproduce
```
CREATE OR REPLACE TABLE jason (j JSON);
INSERT INTO jason VALUES
('{ "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] }'),;
CREATE OR REPLACE TABLE farchar (i INT, d DATE, v VARCHAR);
INSERT INTO farchar VALUES
(1, '2025-01-01'::date, 'anatidae'),;
```
This first case fails as expected because we compare VARCHAR to a JSON, but for some reason comparisons against other types do not fail (see below)
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = (jason.j -> '$.family');
Conversion Error:
Malformed JSON at byte 0 of input: unexpected character. Input: "anatidae" when casting from source column v
LINE 4: ON farchar.v = (jason.j -> '$.family');
```
This query succeeds but returns unexpected results because double quotes are preserved in the cast to VARCHAR
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = (jason.j -> '$.family')::VARCHAR;
┌──────────┬──────┐
│ v │ j │
│ varchar │ json │
├──────────┼──────┤
│ anatidae │ NULL │
└──────────┴──────┘
```
Here is the workaround, either works.
```
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = TRIM((jason.j -> '$.family'), '"');
┌──────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ v │ j │
│ varchar │ json │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ anatidae │ { "created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└──────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
D SELECT *
FROM farchar
LEFT JOIN jason
ON farchar.v = TRIM((jason.j -> '$.family')::VARCHAR, '"');
┌──────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ v │ j │
│ varchar │ json │
├──────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ anatidae │ { "created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└──────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘
```
## Other types work!
Now interestingly comparisons against anything else other than VARCHARs will work...
A date:
```
D SELECT
*
FROM farchar
LEFT JOIN jason
ON farchar.d = (jason.j -> '$.created_at');
┌───────┬────────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ i │ d │ v │ j │
│ int32 │ date │ varchar │ json │
├───────┼────────────┼──────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ 2025-01-01 │ anatidae │ { "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└───────┴────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
An int:
```
D SELECT
*
FROM farchar
LEFT JOIN jason
ON farchar.i = (jason.j -> '$.id');
┌───────┬────────────┬──────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ i │ d │ v │ j │
│ int32 │ date │ varchar │ json │
├───────┼────────────┼──────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ 1 │ 2025-01-01 │ anatidae │ { "id":"1","created_at":"2025-01-01", "family": "anatidae", "species": [ "duck", "goose", "swan", null ] } │
└───────┴────────────┴──────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
```
### OS:
MacOS
### DuckDB Version:
1.4.2
### DuckDB Client:
Python
### Hardware:
_No response_
### Full Name:
Kyle Cheung
### Affiliation:
Greybeam
### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?
- [x] Yes, I have
### Did you include all code required to reproduce the issue?
- [x] Yes, I have
### Did you include all relevant data sets for reproducing the issue?
Yes
Contributor guide
Assessment
This issue has not been assessed yet.