[FEATURE REQUEST]: GetNumPartitions
- Dominant language
- C#
- Stars
- 2.1k
- Forks
- 332
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 9
Description
**Is your feature request related to a problem? Please describe.**
It takes a lot of effort to detect the number of partitions in a dataframe. Currently I have to swap back and forth between the spark U/I and my driver code that is executing in the debugger. It seems like there should be an easy way to show the number of partitions, possibly with the V.S. debugger watch window.
**Describe the solution you'd like**
It would be nice to have a simple method, GetNumPartitions. Perhaps it could be done in an extension class, if it would be confusing to added it to DataFrame directly (ie. perhaps the goals for this class discourage us from polluting it with methods that we don't find in Python/Scala).
**Describe alternatives you've considered**
In Python and Scala these types of things have been handled by relying on the old RDD interface. See https://stackoverflow.com/questions/42171499/get-current-number-of-partitions-of-a-dataframe
I considered using reflection to try to interact with the "internal" RDD implementation that was previously available in this github project. I'd rather not go too far off the beaten path and, if this type of approach was worthwhile, then I think someone would have added it already (maybe even into a "PreviewFeatures.RddExtensions" namespace or something along those lines). Can someone tell me if there is some reasonable way to bring back certain aspects of RDD, until dataframe has reached the same level of functionality? Note that python and scala both give us full access to ".rdd" and I think it is unfair that **.Net for Spark** is so strict in disallowing those features. (another related thing that we might be able to accomplish with a ".rdd" member is to redefine the schema of the dataframe).
**Additional context**
There have been other issues that would have been easier to solve with the help of ".rdd" methods.
As a matter of curiosity, **is it common for developers to use reflection for accessing all the RDD functionality that way**? I suppose I would be more willing to take that type of an approach if everyone else is already doing it. But it seems like a bit of a crutch, and it seems like it is being deliberately discouraged.
Contributor guide
Assessment
This issue has not been assessed yet.