dotnet / dotnet/spark

[FEATURE REQUEST]: Add support or documentation/examples on how to connect to common Azure services

Open
#287 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

One of the advantages of using Spark.NET over Scala/Java/Python is that you get the ecosystem benefits of coding in C# with VisualStudio as well as how this ecosystem works very well hand in hand with Azure. Most C# developers will use Azure for Cloud Services, so it seems to me that Spark.NET should have a great experience when reading/writing data from Azure Cloud services (like EventHub, CosmosDB, Azure Sql, Azure Storage, etc.).

The current way to do this using Spark.NET is that you need to pass in an extra argument (--jars) when running spark-submit passing Java libraries that add this support, and it is only limited to reading and writing where you manually need to pass in the connection configuration. These Java libraries have other functionalities built in, like generating connection configurations in order to connect to the Azure service, as well as other functionalities specific to each Azure service which you can't use through Spark.NET.

This issue is to track either the work of adding native support for connecting to Azure services without the need to pass in extra --jars, or to add examples and documentation to this repo on how to connect to them today.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how spark-submit currently uses the --jars argument and how Azure services such as EventHub, CosmosDB, Azure SQL, and Azure Storage are accessed today. The issue names no files or tests, and its scope must first be narrowed to native support or documentation/examples. Done would mean either removing the extra-jars requirement or providing working guidance for the listed services.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp, spark
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.