stackabletech / stackabletech/trino-operator

Support Delta connector

Open
#538 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
63
Forks
13
Avg merge
1d 20h
Merged PRs (30d)
12

Description

Docs: https://trino.io/docs/current/connector/delta-lake.html

I was able to setup a delta connector with the following workaround 👇🏻 :

  1. Established a TrinoCatalog using Hive as the connector to add S3 connection capabilities, necessary for accessing Delta tables stored in S3.
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
  name: hive
  namespace: infrastructure
  labels:
    trino: trino
spec:
  connector:
    hive:
      metastore:
        configMap: hive
      s3:
        reference: spark-s3-connection
  configOverrides:
    hive.metastore.username: delta

This allows me to add the s3 connection capabilities.

  1. Configured a ConfigMap named trino-delta-properties with settings for the Delta connector. This configuration mirrors the Hive setup but excludes hive.security=allow-all due to compatibility issues. It also introduces a Delta-specific setting to enable table registration from S3 storage.
apiVersion: v1
kind: ConfigMap
metadata:
  name: trino-delta-properties
  namespace: infrastructure
data:
  delta.properties: |
    connector.name=delta_lake
    hive.metastore.uri=thrift://hive.default.svc.cluster.local:9083
    hive.metastore.username=delta
    hive.s3.aws-access-key=${ENV:CATALOG_HIVE_HIVE_S3_AWS_ACCESS_KEY}
    hive.s3.aws-secret-key=${ENV:CATALOG_HIVE_HIVE_S3_AWS_SECRET_KEY}
    hive.s3.endpoint=http://s3
    hive.s3.path-style-access=true
    hive.s3.ssl.enabled=false
    delta.register-table-procedure.enabled=true

this configuration has an additional parameter delta.register-table-procedure.enabled=true as it'srequired in order to register delta tables that are already in the s3 object storage.

  1. Adjusted podOverrides for both worker and coordinator to mount the ConfigMap. However, this method replaces the entire /stackable/config/catalog directory, inadvertently removing the hive.properties file generated by TrinoCatalog. While sufficient for some scenarios, this approach may not cater to all use cases due to its overriding behavior.

Alternatively use initContainers to inject the Delta connector properties into the existing configuration directory without overwriting existing configurations.

podOverrides:
     spec:
        containers:
        - name: trino
           volumeMounts:
           - name: delta-properties
             mountPath: /stackable/config/catalog
        volumes:
        - name: access-control-properties
          configMap:
          - name: delta-properties
             configMap:
               name: trino-delta-properties
    roleGroups:
      default:
        replicas: 1

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Trino Delta connector documentation and the TrinoCatalog, ConfigMap, podOverrides, and initContainers configuration described in the issue. Determine how Delta properties can be supported without replacing generated catalog files, and consider the work complete when Delta tables can be configured without the documented workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.