apache / apache/airavata-data-catalog

Investigate free form queries against the metadata column

Open
#27 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
10
Forks
6
PR merge metrics
No merged PRs in 30d

Description

The Calcite integration only support queries against pre-registered metadata schema fields, like so
```sql
SELECT
*
FROM
my_schema
WHERE
(
field1 < 5
OR field3 = 'bar'
)
AND field1 > 0
AND external_id = 'fff';
```

But it would be nice if one could query against unregistered metadata schema fields that are known to exist within the metadata JSONB column, something like:
```sql
SELECT
*
FROM
my_schema
WHERE
metadata.some_other_field > 0;
```

There are two challenges. One is how to relax Calcite's validation to allow referencing fields that aren't known ahead of time. Second is how to support a syntax for referencing a JSON field that Calcite will parse.

One option might be to have the client queries use JSON functions that Calcite supports: https://calcite.apache.org/docs/reference.html#json-functions

For example:
```sql
SELECT
*
FROM
my_schema
WHERE
JSON_EXISTS(metadata, '$.some_other_field > 0');
```

But, PostgreSQL doesn't yet natively support these functions (see https://www.depesz.com/2022/04/01/waiting-for-postgresql-15-sql-json-query-functions/) so they would need to be rewritten.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the Calcite integration's query-validation path and the metadata JSONB translation boundary; no repository files or tests are named. Compare Calcite's supported JSON functions with PostgreSQL's available operators, then define the query syntax, validation behavior, rewrite strategy, and verification needed for registered and unregistered fields.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql, sql
Domain
backend-api-design, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.