Databricks sync issues

First Stop

• I have read and agreed to the ThoughtSpot LICENSE.
• I understand that if I have a critical issue with ThoughtSpot, I should reach out to my designated Account Executive and Customer Success Manager instead of cs_tools.

Platform Configuration

Description

I'm trying to setup the Databricks syncer and having some issues.

1. It looks like you check the access token for 'dapi' prefix. This is true for personal access tokens, but not service principals which start with 'dose'. Does this work with service principals?

2. I get delta_failed_to_merge errors whenever I try to run a syncer. This happens for both bi_server and metadata. Example below

[DELTA_FAILED_TO_MERGE_FIELDS] Failed to merge fields 'response_size' and 'response_size'                             
                        [SQL:                                                                                                                                                                          
                                    COPY INTO ts_bi_server                                                                                                                                             
                                    FROM 'dbfs:/FileStore/cs_tools/TMP_STAGE_ts_bi_server_6f053.csv'                                                                                                   
                                    FILEFORMAT = CSV                                                                                                                                                   
                                    FORMAT_OPTIONS (                                                                                                                                                   
                                        'header' = 'true',                                                                                                                                             
                                        'inferSchema' = 'true',                                                                                                                                        
                                        'delimiter' = '|'                                                                                                                                              
                                    )                                                                                                                                                                  
                                    COPY_OPTIONS ('mergeSchema' = 'true')                                                                                                                              
                                    ]                                                                                                                                                                  
                        (Background on this error at: https://sqlalche.me/e/20/4xp6)                                                                                                                   

Switching the code to overwriteSchema seems to work

SQL_COPY_INTO = sa.sql.text(
            f"""
            COPY INTO {into}
            FROM 'dbfs:/FileStore/cs_tools/{from_}'
            FILEFORMAT = CSV
            FORMAT_OPTIONS (
                'header' = 'true',
                'inferSchema' = 'true',
                'delimiter' = '|'
            )
            COPY_OPTIONS ('overwriteSchema' = 'true', 'OVERWRITE' = 'true')
            """
        )

Considering adding a log file.

No response

No file or test is named. Start by locating the Databricks syncer and the SQL_COPY_INTO code shown in the report, then reproduce the service-principal token check and DELTA_FAILED_TO_MERGE_FIELDS error for bi_server or metadata. Done means service-principal tokens are accepted and syncs complete without the reported schema-merge failure.