Azure / Azure/azure-sdk-for-java
[QUERY] How to use Linked Services with Cosmos DB Spark connector Throughput control in Synapse Analytics
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 2.2k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 178
Description
**Query/Question**
What is the equivalent to `spark.synapse.linkedService` for use with `spark.cosmos.throughputControl.globalControl.database`
I need to use private endpoints because of DEP, and having credentials stored with Linked Services is very handy.
***Why is this not a Bug or a feature Request?***
Feels like I'm missing something really obvious, but the [docs](https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/cosmos/azure-cosmos-spark_3_2-12/docs/configuration-reference.md) don't seem to cover what I'm seeking,
**Setup (please complete the following information if applicable):**
- OS: Spark 3.4 Python 3.10 Scala
- IDE: Synapse Analytics Web based
- Library/Libraries: default Spark Pool config
**PySpark**
```python
(
df_floodre
.withColumn('id', F.col('uprn').cast('string'))
.write
.format("cosmos.oltp")
.mode("Append")
.option("spark.synapse.linkedService", "CosmosDB_GraphQL_databasets")
.option("spark.cosmos.container", 'floodRe')
.option('spark.cosmos.throughputControl.enabled','true')
.option('spark.cosmos.throughputControl.name','floodRe_Loader_Jobs_Group')
.option('spark.cosmos.throughputControl.globalControl.database','CosmosDB_GraphQL_databasets_ThroughputControl')
.option('spark.cosmos.throughputControl.globalControl.container','ThroughputControl-floodre')
.option('spark.cosmos.throughputControl.targetThroughput',7000)
.save()
)
```
I've tried a few likely nonsensical permutations and got 403 or 404 errors. Guidance much appreciated.
Contributor guide
Assessment
This issue has not been assessed yet.