databrickslabs/sdp-meta

View on GitHub

Metadata driven Spark Declarative Pipelines framework for bronze/silver pipelines

Stars: 285
Forks: 137
Open beginner issues: 0
Indexed issues: 30
Avg merge: 1d 16h
Merged PRs (30d): 49

Dominant language: Python
License: No license data
Last GitHub push: Sep 15, 2026
Latest indexed: Sep 20, 2026
Contributing guide: No contributing guide
Code of conduct: No code of conduct
Beginner labels: help wanted

Beginner friendly Everything

Open All

30 open issues indexed

Document v0.1.1 compatibility boundaries and semantic changes Open

documentation

databrickslabs/sdp-meta#461 · 1 comment · 1 assignee · Opened Sep 18, 2026
Make DAB validation compatible with customized existing bundles Open

bug

databrickslabs/sdp-meta#459 · 2 comments · 1 assignee · Opened Sep 18, 2026
Raise and test the minimum supported Databricks SDK Open

bug

databrickslabs/sdp-meta#457 · 1 assignee · Opened Sep 18, 2026
v0.1.1 Release plan Open

release

databrickslabs/sdp-meta#453 · 1 assignee · Opened Sep 17, 2026
Add Optional Native SDP SQL Gold Layer Support Open

enhancement

databrickslabs/sdp-meta#448 · 1 assignee · Opened Sep 16, 2026
Replace deprecated serverless client field with environment_version Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 35/100

databrickslabs/sdp-meta#437 · Opened Sep 14, 2026
Quality engines: correct native quarantine semantics, opt-in Lakeflow engine, plugin SPI, optional DQX Open

enhancement

databrickslabs/sdp-meta#435 · 9 comments · 1 assignee · Opened Sep 9, 2026
Remove CodeQL github action Open

release

databrickslabs/sdp-meta#424 · 1 assignee · Opened Sep 2, 2026
Support Lakeflow connect (LFC) with Postgres Open

enhancement

databrickslabs/sdp-meta#371 · 1 assignee · Opened Jul 27, 2026
Wrap all error messages in databricks app with custom messages Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

databrickslabs/sdp-meta#369 · Opened Jul 22, 2026
add system_sequence_by attribute support for create_auto_cdc_flow Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

databrickslabs/sdp-meta#359 · 2 comments · Opened Jul 15, 2026
Add spark conf attribute to support create_streaming_table, create_auto_cdc_flow, append_flow apis Open

enhancement

Difficulty 3/5 1-2 days Newbie friendliness 45/100

databrickslabs/sdp-meta#358 · 1 comment · Opened Jul 15, 2026
Support for Historical CDC Backfill (Append Flow with once=True) for Non-Delta Sources (e.g., Parquet) in DLT-META Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

databrickslabs/sdp-meta#271 · Opened Mar 23, 2026
Feature Request: Optional App of the DLT-Meta Jobs for easier observability/tracking Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

databrickslabs/sdp-meta#270 · Opened Mar 20, 2026
Feature Request: Add Tagging as part of the onboarding process Open

enhancement

databrickslabs/sdp-meta#269 · 1 comment · 1 assignee · Opened Mar 20, 2026
Feature request: allow a custom `next_snapshot_and_version` lambda to override the built-in `snapshot_format: "delta"` view path Open

bug

databrickslabs/sdp-meta#268 · 1 assignee · Opened Mar 4, 2026
Bug: `apply_changes_from_snapshot` raises "Snapshot reader function not provided!" when `snapshot_format: "delta"` is configured Open

bug

databrickslabs/sdp-meta#266 · 1 assignee · Opened Mar 4, 2026
CDC (insert/update/delete) flows could not use Data Quality Expectations (DQE) Open

enhancement

databrickslabs/sdp-meta#265 · 1 comment · 1 assignee · Opened Mar 3, 2026
Add option to specify schema for silver tables Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 15/100

databrickslabs/sdp-meta#264 · Opened Feb 23, 2026
Added column comments example to custom transform notebook and docs Open

Difficulty 2/5 1-3 hours Newbie friendliness 15/100

databrickslabs/sdp-meta#258 · Opened Feb 11, 2026
Add support for ForEachBatch sinks Open

enhancement

Difficulty 4/5 3-5 days Newbie friendliness 35/100

databrickslabs/sdp-meta#256 · Opened Feb 11, 2026
Add support for spark declarative pipelines Open

enhancement

databrickslabs/sdp-meta#255 · 1 comment · 1 assignee · Opened Feb 6, 2026
Support for Multiple Columns in cluster_by in DLT Meta Framework Open

help wanted

databrickslabs/sdp-meta#253 · 2 comments · 1 assignee · Opened Jan 16, 2026
Support for Multiple Columns in sequence_by in DLT Meta Framework Open

Difficulty 3/5 1-2 days Newbie friendliness 45/100

databrickslabs/sdp-meta#252 · 2 comments · Opened Jan 16, 2026
Bronze Tables Not Created in Onboarding-Specified Schema Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

databrickslabs/sdp-meta#250 · 3 comments · Opened Dec 30, 2025
Check feasibility of running just silver layer pointing to source Open

enhancement

databrickslabs/sdp-meta#242 · 1 assignee · Opened Nov 17, 2025
Provide the option for Materialized Views for Silver - not just Streaming Tables Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

databrickslabs/sdp-meta#119 · 1 comment · Opened Nov 11, 2024
Create TPS Gen merge Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 20/100

databrickslabs/sdp-meta#102 · Opened Oct 10, 2024
provide SQL Query support Open

enhancement

Difficulty 5/5 Over a week Newbie friendliness 25/100

databrickslabs/sdp-meta#88 · 1 comment · Opened Aug 12, 2024
Point DLT-META to Source System using lakeflow connect Open

enhancement

databrickslabs/sdp-meta#55 · 1 assignee · Opened May 9, 2024

databrickslabs/sdp-meta

Get new issues in your inbox