microsoft / microsoft/DacFx

DW - Publish is too slow to large Warehouse

Open
#794 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug performance sqldw
Dominant language
C#
Stars
460
Forks
29
Avg merge
4d 9h
Merged PRs (30d)
7

Description

  • SqlPackage Version: 170.3.93 & 170.4.80-preview
  • .NET Framework (Windows-only) or .NET Core: N/A
  • Environment (local platform and source/target platforms): Windows / Linux

Our Fabric Warehouse release pipeline is significantly delayed when running SqlPackage with /Action:Publish. The delay appears to happen during DacFx target schema/model loading, during publish planning.

The slow part appears to be a metadata query emitted by DacFx that self-joins sys.tables on history_table_id:

Query
SELECT * FROM (
SELECT  
        [t].[schema_id]              AS [SchemaId],
        SCHEMA_NAME([t].[schema_id]) AS [SchemaName],
        [t].[name]                   AS [ColumnSourceName], 
        [t].[object_id]              AS [TableId],
        [t].[type]                   AS [Type],
        [ds].[type]                  AS [DataspaceType],
        [ds].[data_space_id]         AS [DataspaceId],
        [ds].[name]                  AS [DataspaceName],
        [si].[index_id]              AS [IndexId],
        [si].[type]                  AS [IndexType],
        CASE WHEN exists(SELECT 1 FROM [sys].[columns] AS [c]  WHERE [c].[object_id] = [st].[object_id] AND [st].[is_memory_optimized] = 0 AND  ([c].[system_type_id] IN (34, 35, 99, 241) OR ( [c].[system_type_id] in (165, 167,231,240) AND [c].[max_length] = -1))) THEN
            [dsx].[data_space_id] 
        ELSE
            NULL
        END AS [TextFilegroupId],
        CASE WHEN [dsx].[name] = 'XTP' THEN NULL ELSE [dsx].[name] END AS [TextFilegroupName],
        CASE WHEN exists(SELECT 1 FROM [sys].[columns] AS [c]  WHERE [c].[object_id] = [st].[object_id] AND  [c].[is_filestream] = 1) THEN
            [dsf].[data_space_id] 
        ELSE
            NULL
        END AS [FileStreamId],
        [dsf].[name]                  AS [FileStreamName],
        [dsf].[type]                  AS [FileStreamType],
        [st].[uses_ansi_nulls]		  AS [IsAnsiNulls],
        CAST(OBJECTPROPERTY([st].[object_id],N'IsQuotedIdentOn') AS bit) AS [IsQuotedIdentifier],
        CAST([st].[lock_on_bulk_load] AS bit) 
                                      AS [IsLockedOnBulkLoad],
        [st].[text_in_row_limit]	  AS [TextInRowLimit],
        CAST([st].[large_value_types_out_of_row] AS bit)
                                      AS [LargeValuesOutOfRow],
        CAST(OBJECTPROPERTY([st].[object_id], N'TableHasVarDecimalStorageFormat') AS bit)
                                      AS [HasVarDecimalStorageFormat],
        [st].[is_tracked_by_cdc]      AS [IsTrackedByCDC],
        CAST(CASE
             WHEN [ctt].[object_id] IS NULL THEN 0
             ELSE 1
             END AS BIT)              AS [IsChangeTrackingOn],
        [ctt].[is_track_columns_updated_on] AS [IsTrackColumnsUpdatedOn],
        [st].[lock_escalation]        AS [LockEscalation],     
        CASE WHEN [st].[is_replicated] <> 0 OR [st].[is_merge_published] <> 0 OR [st].[is_schema_published] <> 0 OR [st].[is_published] <> 0 THEN 1 ELSE 0 END AS [ReplInfo]        ,
        [t].[create_date] AS [CreateDate],
        CAST([st].[is_memory_optimized] AS BIT) AS [IsMemoryOptimized],
        [st].[durability] AS [Durability],
        [st].[temporal_type] AS [TemporalType],
        [historyTable].[name] AS [HistoryTableName],
        SCHEMA_NAME([historyTable].[schema_id]) AS [HistoryTableSchema],
        [st].[history_retention_period] AS [HistoryRetentionPeriod],
        [st].[history_retention_period_unit] AS [HistoryRetentionUnit],
        [st].[is_node] AS IsNode,
        [st].[is_edge] AS IsEdge,
        [st].[ledger_type]                              AS [LedgerType],
        [st].[is_dropped_ledger_table]                  AS [IsDroppedLedgerCurrentTable],
        [ledgerView].[name]                             AS [LedgerViewName],
        SCHEMA_NAME([ledgerView].[schema_id])           AS [LedgerViewSchemaName],
        [lvtic].[name]                                  AS [LedgerViewTransactionIdColumnName],
        [lvsnc].[name]                                  AS [LedgerViewSequenceNumberColumnName],
        [lvotc].[name]                                  AS [LedgerViewOperationTypeColumnName],
        [lvotdc].[name]                                 AS [LedgerViewOperationTypeDescColumnName],
        [ledgerCurrentTable].[is_dropped_ledger_table]  AS [IsDroppedLedgerHistoryTable]
FROM    
        [sys].[objects] [t] 
        LEFT    JOIN [sys].[tables] [st]  ON [t].[object_id] = [st].[object_id]
        LEFT    JOIN (SELECT * FROM [sys].[indexes]  WHERE ISNULL([index_id],0) < 2) [si] ON [si].[object_id] = [st].[object_id]
        LEFT    JOIN [sys].[data_spaces] [ds]  ON [ds].[data_space_id] = [si].[data_space_id]
        LEFT    JOIN [sys].[data_spaces] [dsx]  ON [dsx].[data_space_id] = [st].[lob_data_space_id]
        LEFT    JOIN [sys].[data_spaces] [dsf]  ON [dsf].[data_space_id] = [st].[filestream_data_space_id]
        LEFT    JOIN [sys].[change_tracking_tables] [ctt]  ON [ctt].[object_id] = [st].[object_id]
        LEFT    JOIN [sys].[periods] [periods]  ON [periods].[object_id] = [st].[object_id]
        LEFT    JOIN [sys].[tables] [historyTable]  ON [st].[history_table_id] = [historyTable].[object_id]
        LEFT    JOIN [sys].[views] [ledgerView]  ON [ledgerView].[object_id] = [st].[ledger_view_id]
        LEFT    JOIN [sys].[columns] [lvtic]  ON [lvtic].[object_id] = [st].[ledger_view_id] AND [lvtic].[ledger_view_column_type] = 1
        LEFT    JOIN [sys].[columns] [lvsnc]  ON [lvsnc].[object_id] = [st].[ledger_view_id] AND [lvsnc].[ledger_view_column_type] = 2
        LEFT    JOIN [sys].[columns] [lvotc]  ON [lvotc].[object_id] = [st].[ledger_view_id] AND [lvotc].[ledger_view_column_type] = 3
        LEFT    JOIN [sys].[columns] [lvotdc]  ON [lvotdc].[object_id] = [st].[ledger_view_id] AND [lvotdc].[ledger_view_column_type] = 4
        LEFT    JOIN [sys].[tables] [ledgerCurrentTable]  ON [ledgerCurrentTable].[history_table_id] = [st].[object_id]
WHERE  [t].[type] = N'U' AND ISNULL([st].[is_filetable],0) = 0 AND ISNULL([st].[is_external],0) = 0 AND ([t].[is_ms_shipped] = 0 AND NOT EXISTS (SELECT *
                                        FROM [sys].[extended_properties] 
                                        WHERE     [major_id] = [t].[object_id]
                                              AND [minor_id] = 0
                                              AND [class] = 1
                                              AND [name] = N'microsoft_database_tools_support'
                                       ))
AND SCHEMA_NAME([t].[schema_id]) <> N'cdc'
) AS [_results] ORDER BY TableId

and particularly this part

LEFT JOIN sys.tables AS ledgerCurrentTable
    ON ledgerCurrentTable.history_table_id = st.object_id

This join is only used to populate:

ledgerCurrentTable.is_dropped_ledger_table AS IsDroppedLedgerHistoryTable

In our Fabric Warehouse this query is slow (~ 5-6 mins ), even though sys.tables only has ~974 rows and there are no rows where history_table_id IS NOT NULL.

Logs:

2026-05-20T09:14:48 : Perf: Operation started (name, details): Top Level Populator,SqlAzureV12TablePopulator
2026-05-20T09:19:53 : Perf: Operation started (name, details): Child Populator,SqlDwUnifiedTableColumnPopulator
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Child Populator,SqlDwUnifiedTableColumnPopulator,1652
2026-05-20T09:19:55 : Perf: Operation started (name, details): Child Populator,SqlDwUnifiedClusterByColumnPopulator
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Child Populator,SqlDwUnifiedClusterByColumnPopulator,0
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Top Level Populator,SqlAzureV12TablePopulator,306838 ms 

Steps to Reproduce:

  1. Create a empty Warehouse.
  2. Add 500-1000 tables to a SDK-style SQL project.
  3. Build dacpac
  4. Run sqlpackage /Action:Publish - Initializing deployment finishes in no time.
  5. Build dacpac
  6. Run sqlpackage /Action:Publish - Initializing deployment takes very long time.

or download my repro containing Fabric Warehouse and corresponding SQL project.

  1. Download and extract repro.zip
  2. Add workspace_id to run.py
  3. Run

Expected:
SqlPackage /Action:Publish should not spend significant time loading Fabric Warehouse metadata when sys.tables has ~974 rows and history_table_id IS NOT NULL returns 0 rows.

Did this occur in prior versions? If not - which version(s) did it work in?

  • (DacFx/SqlPackage/SSMS/Azure Data Studio)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the SqlPackage /Action:Publish entry point and the DacFx target schema/model loading described in the issue. Reproduce the delay with the linked repro.zip and run.py against a Fabric Warehouse containing 500–1000 tables, then inspect the metadata query and its history_table_id and ledgerCurrentTable join. Done means publish no longer spends several minutes loading metadata when history_table_id has no matching rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.