ClickHouse / ClickHouse/pg_clickhouse

Support for ClickHouse Map type in pg_clickhouse

Open
#128 4 comments 1 reaction 1 assignee Claimed by @serprex View on GitHub
data types enhancement
Dominant language
C
Stars
283
Forks
21
Avg merge
2d 6h
Merged PRs (30d)
11

Description

## Issue Type

- [x] Feature Request
- [ ] Bug Report
- [ ] Documentation Issue

## Description

When using pg_clickhouse to query ClickHouse tables with `Map` type columns, the FDW converts the Map type to PostgreSQL `text` type. This makes it impossible to efficiently query Map fields and filter by specific keys/values.

## Current Behavior

Given a ClickHouse table with a Map column:

```sql
CREATE TABLE default.traces (
id UUID,
name String,
metadata Map(LowCardinality(String), String)
) ENGINE = MergeTree() ORDER BY id;
```

When importing this table into PostgreSQL via pg_clickhouse:

```sql
CREATE FOREIGN TABLE langfuse.traces (
id text,
name text,
metadata text -- Map is converted to text
)
SERVER clickhouse_server
OPTIONS (database 'default', table_name 'traces');
```

The `metadata` column becomes `text` type in PostgreSQL, containing the string representation like `{'status':'idle'}`.

### Limitations

1. **Cannot use Map functions**: ClickHouse's native Map functions (`mapContains`, `mapGet`, `metadata['key']`) cannot be used because PostgreSQL treats the column as `text`

2. **Cannot use string functions**: String functions (`LIKE`, `position`, `substring`) fail when pushed down to ClickHouse because ClickHouse rejects them for Map types

3. **Cannot use PostgreSQL JSONB functions**: JSONB functions (`::jsonb->>'key'`) are pushed down to ClickHouse, which doesn't support PostgreSQL's JSONB functions

### Example Failures

```sql
-- ClickHouse Map functions don't work
SELECT metadata['status'] FROM langfuse.traces;
-- ERROR: cannot subscript type text because it does not support subscripting

-- String functions fail when pushed down
SELECT * FROM langfuse.traces WHERE metadata LIKE '%status%';
-- ERROR: pg_clickhouse: Code: 43. DB::Exception: Illegal type Map(String, String) of argument of function like

-- PostgreSQL JSONB functions fail
SELECT metadata::jsonb->>'status' FROM langfuse.traces;
-- Works only with LIMIT (prevents pushdown), not efficient for large datasets
```

## Desired Behavior

We need a way to query Map columns efficiently from PostgreSQL, with filter pushdown to ClickHouse.

### Potential Solutions

#### Option 1: Map to JSONB Type (Recommended)

Automatically map ClickHouse `Map` type to PostgreSQL `JSONB` type:

```sql
-- Automatic type mapping
CREATE FOREIGN TABLE langfuse.traces (
id text,
metadata jsonb -- Map mapped to JSONB
)
SERVER clickhouse_server
OPTIONS (database 'default', table_name 'traces');

-- Now we can query:
SELECT metadata->>'status' FROM langfuse.traces;
-- This should push down to ClickHouse as: mapContains(metadata, 'status')
```

This would involve:
1. Detecting Map types during import
2. Converting Map to JSON format when querying
3. Translating PostgreSQL JSONB operations to ClickHouse Map operations

#### Option 2: Pass-through Map Type

Support a special PostgreSQL type (e.g., `clickhouse_map`) that preserves the Map type:

```sql
-- Explicit type specification
CREATE FOREIGN TABLE langfuse.traces (
id text,
metadata clickhouse_map -- Preserve Map type
)
SERVER clickhouse_server
OPTIONS (database 'default', table_name 'traces');

-- Use Map functions directly
SELECT mapContains(metadata, 'status') FROM langfuse.traces;
```

#### Option 3: Column-Level Type Options

Allow specifying column-level type options:

```sql
CREATE FOREIGN TABLE langfuse.traces (
id text,
metadata text OPTIONS (clickhouse_type 'map')
)
SERVER clickhouse_server
OPTIONS (database 'default', table_name 'traces');
```

## Use Case

This feature is essential for applications that:

1. Store metadata as flexible key-value pairs in ClickHouse Map columns
2. Need to query and filter by specific metadata fields from PostgreSQL
3. Require high performance with filter pushdown (not post-processing in PostgreSQL)
4. Cannot modify the ClickHouse table structure (e.g., cannot add additional columns)

### Example Scenario

We have a LLM tracing system that stores traces in ClickHouse:

```sql
-- ClickHouse table
CREATE TABLE traces (
id UUID,
name String,
model String,
metadata Map(String, String) -- Stores: {'status': 'pending', 'version': '1.0', 'user_id': '123'}
) ENGINE = MergeTree() ORDER BY id;
```

Common queries needed from PostgreSQL:

```sql
-- Find all traces with status = 'completed'
SELECT * FROM traces WHERE metadata->>'status' = 'completed';

-- Count traces by status
SELECT metadata->>'status', COUNT(*) FROM traces GROUP BY metadata->>'status';

-- Filter by multiple metadata fields
SELECT * FROM traces
WHERE metadata->>'status' = 'completed'
AND metadata->>'version' = '1.0';
```

Currently, these queries either fail or require fetching all data and filtering in PostgreSQL, which is not scalable for large datasets.

## Environment

- **ClickHouse Version**: 24.12.3.47 (official build)
- **PostgreSQL Version**: PostgreSQL 15.x
- **pg_clickhouse Version**: Latest (unknown exact version)
- **OS**: Linux

## Additional Context

### Why Not Use Alternatives?

1. **Cannot modify ClickHouse table**: Adding a JSON column or creating views is not an option in our deployment
2. **Need PostgreSQL interface**: Our application code uses PostgreSQL, not ClickHouse directly
3. **Performance requirements**: Need filter pushdown for large-scale queries (millions of rows)

### Similar Issues

I noticed that PR #74 added support for JSON type mapping to JSONB. This feature request is similar but for Map type, which is the primary JSON-like type used in ClickHouse.

### Workaround Limitations

Current workarounds have significant limitations:

1. **Create View in ClickHouse**: Not allowed in our scenario
2. **Use LIMIT + application filtering**: Not efficient for large datasets, no filter pushdown
3. **Direct ClickHouse connection**: Requires significant code changes, loses PostgreSQL benefits

## References

- PR #74: Add JSONB support (similar pattern, but for JSON type instead of Map)
- ClickHouse Map documentation: https://clickhouse.com/docs/en/sql-reference/data-types/map/
- pg_clickhouse type mapping documentation

## Suggested Implementation

For Option 1 (Map to JSONB), the implementation could:

1. **Type Detection**: During table import, detect Map columns
2. **Automatic Mapping**: Assign JSONB type to Map columns
3. **Query Translation**:
- `jsonb->>'key'` → `mapContains(metadata, 'key') AND metadata['key']`
- `jsonb ? 'key'` → `mapContains(metadata, 'key')`
- `jsonb @> '{"key":"value"}'` → `mapContains(metadata, 'key') AND metadata['key'] = 'value'`
4. **Format Conversion**: Convert Map format (`{'key':'value'}`) to JSON format (`{"key":"value"}`) on read

This would be similar to how PR #74 handled JSON types but tailored for Map types.

## Priority

**High** - This is a critical feature for applications using ClickHouse Map columns with pg_clickhouse FDW.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.