DW - Publish is too slow to large Warehouse
Nobody has claimed this yet.
- Dominant language
- C#
- Stars
- 460
- Forks
- 29
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 7
Description
- SqlPackage Version: 170.3.93 & 170.4.80-preview
- .NET Framework (Windows-only) or .NET Core: N/A
- Environment (local platform and source/target platforms): Windows / Linux
Our Fabric Warehouse release pipeline is significantly delayed when running SqlPackage with /Action:Publish. The delay appears to happen during DacFx target schema/model loading, during publish planning.
The slow part appears to be a metadata query emitted by DacFx that self-joins sys.tables on history_table_id:
Query
SELECT * FROM (
SELECT
[t].[schema_id] AS [SchemaId],
SCHEMA_NAME([t].[schema_id]) AS [SchemaName],
[t].[name] AS [ColumnSourceName],
[t].[object_id] AS [TableId],
[t].[type] AS [Type],
[ds].[type] AS [DataspaceType],
[ds].[data_space_id] AS [DataspaceId],
[ds].[name] AS [DataspaceName],
[si].[index_id] AS [IndexId],
[si].[type] AS [IndexType],
CASE WHEN exists(SELECT 1 FROM [sys].[columns] AS [c] WHERE [c].[object_id] = [st].[object_id] AND [st].[is_memory_optimized] = 0 AND ([c].[system_type_id] IN (34, 35, 99, 241) OR ( [c].[system_type_id] in (165, 167,231,240) AND [c].[max_length] = -1))) THEN
[dsx].[data_space_id]
ELSE
NULL
END AS [TextFilegroupId],
CASE WHEN [dsx].[name] = 'XTP' THEN NULL ELSE [dsx].[name] END AS [TextFilegroupName],
CASE WHEN exists(SELECT 1 FROM [sys].[columns] AS [c] WHERE [c].[object_id] = [st].[object_id] AND [c].[is_filestream] = 1) THEN
[dsf].[data_space_id]
ELSE
NULL
END AS [FileStreamId],
[dsf].[name] AS [FileStreamName],
[dsf].[type] AS [FileStreamType],
[st].[uses_ansi_nulls] AS [IsAnsiNulls],
CAST(OBJECTPROPERTY([st].[object_id],N'IsQuotedIdentOn') AS bit) AS [IsQuotedIdentifier],
CAST([st].[lock_on_bulk_load] AS bit)
AS [IsLockedOnBulkLoad],
[st].[text_in_row_limit] AS [TextInRowLimit],
CAST([st].[large_value_types_out_of_row] AS bit)
AS [LargeValuesOutOfRow],
CAST(OBJECTPROPERTY([st].[object_id], N'TableHasVarDecimalStorageFormat') AS bit)
AS [HasVarDecimalStorageFormat],
[st].[is_tracked_by_cdc] AS [IsTrackedByCDC],
CAST(CASE
WHEN [ctt].[object_id] IS NULL THEN 0
ELSE 1
END AS BIT) AS [IsChangeTrackingOn],
[ctt].[is_track_columns_updated_on] AS [IsTrackColumnsUpdatedOn],
[st].[lock_escalation] AS [LockEscalation],
CASE WHEN [st].[is_replicated] <> 0 OR [st].[is_merge_published] <> 0 OR [st].[is_schema_published] <> 0 OR [st].[is_published] <> 0 THEN 1 ELSE 0 END AS [ReplInfo] ,
[t].[create_date] AS [CreateDate],
CAST([st].[is_memory_optimized] AS BIT) AS [IsMemoryOptimized],
[st].[durability] AS [Durability],
[st].[temporal_type] AS [TemporalType],
[historyTable].[name] AS [HistoryTableName],
SCHEMA_NAME([historyTable].[schema_id]) AS [HistoryTableSchema],
[st].[history_retention_period] AS [HistoryRetentionPeriod],
[st].[history_retention_period_unit] AS [HistoryRetentionUnit],
[st].[is_node] AS IsNode,
[st].[is_edge] AS IsEdge,
[st].[ledger_type] AS [LedgerType],
[st].[is_dropped_ledger_table] AS [IsDroppedLedgerCurrentTable],
[ledgerView].[name] AS [LedgerViewName],
SCHEMA_NAME([ledgerView].[schema_id]) AS [LedgerViewSchemaName],
[lvtic].[name] AS [LedgerViewTransactionIdColumnName],
[lvsnc].[name] AS [LedgerViewSequenceNumberColumnName],
[lvotc].[name] AS [LedgerViewOperationTypeColumnName],
[lvotdc].[name] AS [LedgerViewOperationTypeDescColumnName],
[ledgerCurrentTable].[is_dropped_ledger_table] AS [IsDroppedLedgerHistoryTable]
FROM
[sys].[objects] [t]
LEFT JOIN [sys].[tables] [st] ON [t].[object_id] = [st].[object_id]
LEFT JOIN (SELECT * FROM [sys].[indexes] WHERE ISNULL([index_id],0) < 2) [si] ON [si].[object_id] = [st].[object_id]
LEFT JOIN [sys].[data_spaces] [ds] ON [ds].[data_space_id] = [si].[data_space_id]
LEFT JOIN [sys].[data_spaces] [dsx] ON [dsx].[data_space_id] = [st].[lob_data_space_id]
LEFT JOIN [sys].[data_spaces] [dsf] ON [dsf].[data_space_id] = [st].[filestream_data_space_id]
LEFT JOIN [sys].[change_tracking_tables] [ctt] ON [ctt].[object_id] = [st].[object_id]
LEFT JOIN [sys].[periods] [periods] ON [periods].[object_id] = [st].[object_id]
LEFT JOIN [sys].[tables] [historyTable] ON [st].[history_table_id] = [historyTable].[object_id]
LEFT JOIN [sys].[views] [ledgerView] ON [ledgerView].[object_id] = [st].[ledger_view_id]
LEFT JOIN [sys].[columns] [lvtic] ON [lvtic].[object_id] = [st].[ledger_view_id] AND [lvtic].[ledger_view_column_type] = 1
LEFT JOIN [sys].[columns] [lvsnc] ON [lvsnc].[object_id] = [st].[ledger_view_id] AND [lvsnc].[ledger_view_column_type] = 2
LEFT JOIN [sys].[columns] [lvotc] ON [lvotc].[object_id] = [st].[ledger_view_id] AND [lvotc].[ledger_view_column_type] = 3
LEFT JOIN [sys].[columns] [lvotdc] ON [lvotdc].[object_id] = [st].[ledger_view_id] AND [lvotdc].[ledger_view_column_type] = 4
LEFT JOIN [sys].[tables] [ledgerCurrentTable] ON [ledgerCurrentTable].[history_table_id] = [st].[object_id]
WHERE [t].[type] = N'U' AND ISNULL([st].[is_filetable],0) = 0 AND ISNULL([st].[is_external],0) = 0 AND ([t].[is_ms_shipped] = 0 AND NOT EXISTS (SELECT *
FROM [sys].[extended_properties]
WHERE [major_id] = [t].[object_id]
AND [minor_id] = 0
AND [class] = 1
AND [name] = N'microsoft_database_tools_support'
))
AND SCHEMA_NAME([t].[schema_id]) <> N'cdc'
) AS [_results] ORDER BY TableId
and particularly this part
LEFT JOIN sys.tables AS ledgerCurrentTable
ON ledgerCurrentTable.history_table_id = st.object_id
This join is only used to populate:
ledgerCurrentTable.is_dropped_ledger_table AS IsDroppedLedgerHistoryTable
In our Fabric Warehouse this query is slow (~ 5-6 mins ), even though sys.tables only has ~974 rows and there are no rows where history_table_id IS NOT NULL.
Logs:
2026-05-20T09:14:48 : Perf: Operation started (name, details): Top Level Populator,SqlAzureV12TablePopulator
2026-05-20T09:19:53 : Perf: Operation started (name, details): Child Populator,SqlDwUnifiedTableColumnPopulator
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Child Populator,SqlDwUnifiedTableColumnPopulator,1652
2026-05-20T09:19:55 : Perf: Operation started (name, details): Child Populator,SqlDwUnifiedClusterByColumnPopulator
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Child Populator,SqlDwUnifiedClusterByColumnPopulator,0
2026-05-20T09:19:55 : Perf: Operation ended (name, details, elapsed in ms): Top Level Populator,SqlAzureV12TablePopulator,306838 ms
Steps to Reproduce:
- Create a empty Warehouse.
- Add 500-1000 tables to a SDK-style SQL project.
- Build dacpac
- Run sqlpackage /Action:Publish -
Initializing deploymentfinishes in no time. - Build dacpac
- Run sqlpackage /Action:Publish -
Initializing deploymenttakes very long time.
or download my repro containing Fabric Warehouse and corresponding SQL project.
- Download and extract repro.zip
- Add workspace_id to run.py
- Run
Expected:
SqlPackage /Action:Publish should not spend significant time loading Fabric Warehouse metadata when sys.tables has ~974 rows and history_table_id IS NOT NULL returns 0 rows.
Did this occur in prior versions? If not - which version(s) did it work in?
- (DacFx/SqlPackage/SSMS/Azure Data Studio)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SqlPackage /Action:Publish entry point and the DacFx target schema/model loading described in the issue. Reproduce the delay with the linked repro.zip and run.py against a Fabric Warehouse containing 500–1000 tables, then inspect the metadata query and its history_table_id and ledgerCurrentTable join. Done means publish no longer spends several minutes loading metadata when history_table_id has no matching rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100