stackabletech / stackabletech/trino-operator
Support Delta connector
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 63
- Forks
- 13
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 12
Description
Docs: https://trino.io/docs/current/connector/delta-lake.html
I was able to setup a delta connector with the following workaround 👇🏻 :
- Established a TrinoCatalog using Hive as the connector to add S3 connection capabilities, necessary for accessing Delta tables stored in S3.
apiVersion: trino.stackable.tech/v1alpha1
kind: TrinoCatalog
metadata:
name: hive
namespace: infrastructure
labels:
trino: trino
spec:
connector:
hive:
metastore:
configMap: hive
s3:
reference: spark-s3-connection
configOverrides:
hive.metastore.username: delta
This allows me to add the s3 connection capabilities.
- Configured a ConfigMap named trino-delta-properties with settings for the Delta connector. This configuration mirrors the Hive setup but excludes hive.security=allow-all due to compatibility issues. It also introduces a Delta-specific setting to enable table registration from S3 storage.
apiVersion: v1
kind: ConfigMap
metadata:
name: trino-delta-properties
namespace: infrastructure
data:
delta.properties: |
connector.name=delta_lake
hive.metastore.uri=thrift://hive.default.svc.cluster.local:9083
hive.metastore.username=delta
hive.s3.aws-access-key=${ENV:CATALOG_HIVE_HIVE_S3_AWS_ACCESS_KEY}
hive.s3.aws-secret-key=${ENV:CATALOG_HIVE_HIVE_S3_AWS_SECRET_KEY}
hive.s3.endpoint=http://s3
hive.s3.path-style-access=true
hive.s3.ssl.enabled=false
delta.register-table-procedure.enabled=true
this configuration has an additional parameter
delta.register-table-procedure.enabled=trueas it'srequired in order to register delta tables that are already in the s3 object storage.
- Adjusted
podOverridesfor both worker and coordinator to mount theConfigMap. However, this method replaces the entire/stackable/config/catalogdirectory, inadvertently removing thehive.propertiesfile generated byTrinoCatalog. While sufficient for some scenarios, this approach may not cater to all use cases due to its overriding behavior.
Alternatively use
initContainersto inject the Delta connector properties into the existing configuration directory without overwriting existing configurations.
podOverrides:
spec:
containers:
- name: trino
volumeMounts:
- name: delta-properties
mountPath: /stackable/config/catalog
volumes:
- name: access-control-properties
configMap:
- name: delta-properties
configMap:
name: trino-delta-properties
roleGroups:
default:
replicas: 1
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the Trino Delta connector documentation and the TrinoCatalog, ConfigMap, podOverrides, and initContainers configuration described in the issue. Determine how Delta properties can be supported without replacing generated catalog files, and consider the work complete when Delta tables can be configured without the documented workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100