dotnet / dotnet/spark

[FEATURE REQUEST]: Deprecate and/or evict Microsoft.Data.Analysis from the Microsoft.Spark assembly

Open
#1,171 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
2.1k
Forks
332
Avg merge
1d 20h
Merged PRs (30d)
9

Description

**Is your feature request related to a problem? Please describe.**

I'd like to deprecate **Microsoft.Data.Analysis** from this project, or at least move it out of **Microsoft.Spark** to a distinct assembly that must be introduced separately into the .Net drivers.

It can remain in the .net worker for now (Microsoft.Spark.Worker)

**Describe the solution you'd like**

I'm tired of doing the following :
```
using FxDataFrame = Microsoft.Data.Analysis.DataFrame;
```

This happens too frequently and it is nuts. This nonsense should not be necessary - especially not for a mission-critical class like **DataFrame**. The DataFrame class is perhaps the most **fundamental** component of Spark. The name should not be overloaded to mean two different things within the same Microsoft.Spark assembly.

It is sort of unconventional, but marking all the Udf's deprecated would still allow that version of DataFrame to be used but it would caution users to stay away.

To take a step further, we will move the related Udf stuff to a distinct assembly (**Microsoft.Spark.Miscellaneous.DataAnalysis**). This miscellaneous assembly would be introduced into Spark projects if developers **really** needed to use the "other DataFrame".

Ultimately this change will avoid confusing new Spark engineers. They are often unable to determine which version of the "DataFrame" is the "right" one. That type of confusion is unnecessary. Unfortunately that confusion is encountered all too quickly because of the proximity between the two types of "DataFrames" that are supported by Microsoft.Spark.

> I'm willing to grant that the DataFrames live in different namespaces and that helps reduce confusion, of course. However a new Spark engineer will find the two classes as soon as they download the github project for the first time. They are likely to believe that the non-Spark version of DataFrame is more "**advanced**" or more "**native**" to .Net or is "**better**" in some way (else why would it be in the Spark project to begin with!) These assumptions are all wrong. I have always regretted using the "Microsoft.Data.Analysis" version of DataFrame whenever I took that path.

**Final Note**
I think Microsoft.Data.Analysis must remain as part of the .net worker for the convenience (and to avoid breaking legacy projects).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.