metabase / metabase/metabase

Error trying to show tables with JSONfield columns from redshift db

Open
#28,564 1 comment 0 reactions 0 assignees View on GitHub
.Backend .Team/SemanticLayer Administration/Metadata & Sync Database/Redshift Priority:P3 Type:New Feature
Dominant language
Clojure
Stars
49.3k
Forks
6.8k
Avg merge
1d 13h
Merged PRs (30d)
653

Description

**Describe the bug**
Metabase throws an error when trying to show tables from a redshift database that contain JSONfield columns.

Raw queries work as intended but you can not do anything visual, like build editor or just show the contents of the table.

**Logs**
```
2023-02-23 08:22:11,609 WARN sync.util :: Error syncing Fields for Table 'Table 1,444 'myapp.integrations''
java.lang.IllegalArgumentException: No matching clause: :redshift
at metabase.driver.sql_jdbc.sync.describe_table$describe_nested_field_columns.invokeStatic(describe_table.clj:402)
at metabase.driver.sql_jdbc.sync.describe_table$describe_nested_field_columns.invoke(describe_table.clj:390)
at metabase.driver.postgres$fn__61906.invokeStatic(postgres.clj:230)
at metabase.driver.postgres$fn__61906.invoke(postgres.clj:227)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at metabase.sync.fetch_metadata$fn__67713$nfc_metadata__67718$fn__67719.invoke(fetch_metadata.clj:33)
at metabase.sync.fetch_metadata$fn__67713$nfc_metadata__67718.invoke(fetch_metadata.clj:28)
at metabase.sync.sync_metadata.fields.fetch_metadata$fn__67933$db_metadata__67938$fn__67939.invoke(fetch_metadata.clj:91)
at metabase.sync.sync_metadata.fields.fetch_metadata$fn__67933$db_metadata__67938.invoke(fetch_metadata.clj:85)
at metabase.sync.sync_metadata.fields$fn__68434$sync_fields_for_table_BANG___68443$fn__68446$fn__68447.invoke(fields.clj:72)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:160)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:153)
at metabase.sync.sync_metadata.fields$fn__68434$sync_fields_for_table_BANG___68443$fn__68446.invoke(fields.clj:71)
at metabase.sync.sync_metadata.fields$fn__68434$sync_fields_for_table_BANG___68443.invoke(fields.clj:65)
at clojure.core$partial$fn__5908.invoke(core.clj:2641)
at clojure.core$map$fn__5935.invoke(core.clj:2770)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.RT.seq(RT.java:535)
at clojure.core$seq__5467.invokeStatic(core.clj:139)
at clojure.core$filter$fn__5962.invoke(core.clj:2826)
at clojure.lang.LazySeq.sval(LazySeq.java:42)
at clojure.lang.LazySeq.seq(LazySeq.java:51)
at clojure.lang.ChunkedCons.chunkedNext(ChunkedCons.java:59)
at clojure.core$chunk_next.invokeStatic(core.clj:710)
at clojure.core$reduce1.invokeStatic(core.clj:944)
at clojure.core$reduce1.invokeStatic(core.clj:936)
at clojure.core$merge_with.invokeStatic(core.clj:3076)
at clojure.core$merge_with.doInvoke(core.clj:3068)
at clojure.lang.RestFn.applyTo(RestFn.java:139)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at metabase.sync.sync_metadata.fields$fn__68472$sync_fields_BANG___68477$fn__68478.invoke(fields.clj:85)
at metabase.sync.sync_metadata.fields$fn__68472$sync_fields_BANG___68477.invoke(fields.clj:77)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:669)
at clojure.core$apply.invoke(core.clj:662)
at metabase.sync.util$fn__42851$run_step_with_metadata__42856$fn__42860$fn__42862.doInvoke(util.clj:388)
at clojure.lang.RestFn.invoke(RestFn.java:397)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_debug_logging.invokeStatic(util.clj:119)
at metabase.sync.util$with_start_and_finish_debug_logging.invoke(util.clj:116)
at metabase.sync.util$fn__42851$run_step_with_metadata__42856$fn__42860.invoke(util.clj:383)
at metabase.sync.util$fn__42851$run_step_with_metadata__42856.invoke(util.clj:378)
at metabase.sync.util$fn__43072$run_sync_operation__43077$fn__43078$fn__43086.invoke(util.clj:495)
at metabase.sync.util$fn__43072$run_sync_operation__43077$fn__43078.invoke(util.clj:493)
at metabase.sync.util$fn__43072$run_sync_operation__43077.invoke(util.clj:487)
at metabase.sync.sync_metadata$fn__70082$sync_db_metadata_BANG___70087$fn__70088$fn__70089.invoke(sync_metadata.clj:52)
at metabase.sync.util$do_with_error_handling.invokeStatic(util.clj:160)
at metabase.sync.util$do_with_error_handling.invoke(util.clj:153)
at clojure.core$partial$fn__5910.invoke(core.clj:2647)
at metabase.driver$fn__33725.invokeStatic(driver.clj:626)
at metabase.driver$fn__33725.invoke(driver.clj:626)
at clojure.lang.MultiFn.invoke(MultiFn.java:239)
at metabase.sync.util$sync_in_context$fn__42772.invoke(util.clj:138)
at metabase.sync.util$with_db_logging_disabled$fn__42769.invoke(util.clj:129)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invokeStatic(util.clj:102)
at metabase.sync.util$with_start_and_finish_logging_STAR_.invoke(util.clj:96)
at metabase.sync.util$with_start_and_finish_logging$fn__42758.invoke(util.clj:114)
at metabase.sync.util$with_sync_events$fn__42753.invoke(util.clj:88)
at metabase.sync.util$with_duplicate_ops_prevented$fn__42744.invoke(util.clj:67)
at metabase.sync.util$do_sync_operation.invokeStatic(util.clj:181)
at metabase.sync.util$do_sync_operation.invoke(util.clj:178)
at metabase.sync.sync_metadata$fn__70082$sync_db_metadata_BANG___70087$fn__70088.invoke(sync_metadata.clj:51)
at metabase.sync.sync_metadata$fn__70082$sync_db_metadata_BANG___70087.invoke(sync_metadata.clj:47)
at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invokeStatic(sync_databases.clj:72)
at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invoke(sync_databases.clj:63)
at metabase.task.sync_databases.SyncAndAnalyzeDatabase.execute(sync_databases.clj:82)
at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
```
**To Reproduce**
Steps to reproduce the behavior (if you can reproduce the bug using the Sample Database, we will find the issue faster):
1. Add a redshift database that contains tables with JSONfield columns
2. Sync it
3. Try to view table through metabase
4. See error

**Expected behavior**
To view and handle the table as every other table.

**Screenshots/videos**
![image](https://user-images.githubusercontent.com/16354616/220878649-2f89e084-9f43-4994-b638-4ca94832327d.png)

**Information about your Metabase Installation:**

{
"browser-info": {
"language": "en-US",
"platform": "Linux x86_64",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36",
"vendor": "Google Inc."
},
"system-info": {
"file.encoding": "UTF-8",
"java.runtime.name": "OpenJDK Runtime Environment",
"java.runtime.version": "11.0.17+8",
"java.vendor": "Eclipse Adoptium",
"java.vendor.url": "https://adoptium.net/",
"java.version": "11.0.17",
"java.vm.name": "OpenJDK 64-Bit Server VM",
"java.vm.version": "11.0.17+8",
"os.name": "Linux",
"os.version": "5.4.188-104.359.amzn2.x86_64",
"user.language": "en",
"user.timezone": "UTC"
},
"metabase-info": {
"databases": [
"postgres",
"redshift",
"h2"
],
"hosting-env": "unknown",
"application-database": "postgres",
"application-database-details": {
"database": {
"name": "PostgreSQL",
"version": "11.16"
},
"jdbc-driver": {
"name": "PostgreSQL JDBC Driver",
"version": "42.5.0"
}
},
"run-mode": "prod",
"version": {
"date": "2023-01-17",
"tag": "v0.45.2",
"branch": "release-x.45.x",
"hash": "819ce29"
},
"settings": {
"report-timezone": null
}
}
}

**Severity**
It is blocking leveraging warehousing through metabase for some teams.

Contributor guide

Open the contributing guide

Research direction

Start in metabase/driver/sql_jdbc/sync/describe_table.clj around line 402, then follow the Redshift-related call through metabase/driver/postgres.clj. Reproduce the metadata sync with a Redshift table containing JSONfield columns and inspect the missing :redshift handling. Done means the table sync completes and the table can be viewed and used visually without this error.

Written by the indexing model from the issue text.

Assessment

Tech stack
clojure
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.