apache / apache/spark-connect-go
Add Support for Additional DataFrame Functionality
- Dominant language
- Go
- Stars
- 252
- Forks
- 53
- PR merge metrics
- No merged PRs in 30d
Description
The DataFrame still has lots of missing functions that need to be implemented. This is the umbrella item for that.
Most of them should be relatively mechanical, but some require some additional work.
|Method|Support|
|---|---|
|`agg`|:white_check_mark:|
|`alias`|:white_check_mark:|
|`approxQuantile`|:white_check_mark:|
|`cache`|:white_check_mark:|
|`checkpoint`|:x:|
|`coalesce`|:white_check_mark:|
|`colRegex`|:white_check_mark: (`OfDFWithRegex`)|
|`collect`|:white_check_mark: |
|`columns`|:white_check_mark:|
|`corr`|:white_check_mark:|
|`count`|:white_check_mark:|
|`cov`|:white_check_mark:|
|`createGlobalTempView`|:white_check_mark:|
|`createOrReplaceGlobalTempView`|:white_check_mark:|
|`createOrReplaceTempView`|:white_check_mark:|
|`createTempView`|:white_check_mark: |
|`crossJoin`|:white_check_mark:|
|`crosstab`|:white_check_mark:|
|`cube`|:white_check_mark:|
|`describe`|:white_check_mark:|
|`distinct`|:white_check_mark:|
|`drop`|:white_check_mark:|
|`dropDuplicates`|:white_check_mark:|
|`dropDuplicatesWithinWatermark`|:x:|
|`dropna`|:white_check_mark:|
|`dtypes`|:x:|
|`exceptAll`|:white_check_mark:|
|`executionInfo`|:x:|
|`explain`|:white_check_mark:|
|`fillna`|:white_check_mark:|
|`filter`|:white_check_mark: |
|`first`|:white_check_mark:|
|`foreach`|:heavy_minus_sign: (needs native UDFs)|
|`foreachPartition`|:heavy_minus_sign: (needs native UDFs)|
|`freqItems`|:white_check_mark:|
|`groupBy`|:white_check_mark: (some restrictions apply)|
|`head`|:white_check_mark:|
|`hint`|:x:|
|`inputFiles`|:x:|
|`intersect`|:white_check_mark:|
|`intersectAll`|:white_check_mark:|
|`isEmpty`|:white_check_mark:|
|`isLocal`|:x:|
|`isStreaming`|:x:|
|`is_cached`|:x:|
|`join`|:white_check_mark:|
|`limit`|:white_check_mark:|
|`localCheckpoint`|:x:|
|`mapInArrow`|:heavy_minus_sign: (needs native UDFs)|
|`mapInPandas`|:heavy_minus_sign: (needs native UDFs)|
|`melt`|:white_check_mark:|
|`mergeInto`|:x:|
|`na`|:white_check_mark:|
|`observe`|:x:|
|`offset`|:white_check_mark:|
|`orderBy`|:white_check_mark:|
|`pandas_api`|:heavy_minus_sign:|
|`persist`|:white_check_mark:|
|`printSchema`|:x:|
|`randomSplit`|:white_check_mark:|
|`rdd`|:heavy_minus_sign:|
|`registerTempTable`|:x:|
|`repartition`|:white_check_mark: |
|`repartitionByRange`|:white_check_mark: |
|`replace`|:white_check_mark:|
|`rollup`|:white_check_mark:|
|`sameSemantics`|:white_check_mark:|
|`sample`|:white_check_mark:|
|`sampleBy`|:x:|
|`schema`|:white_check_mark:|
|`select`|:white_check_mark:|
|`selectExpr`|:white_check_mark:|
|`semanticHash`|:white_check_mark:|
|`show`|:white_check_mark: |
|`sort`|:white_check_mark:|
|`sortWithinPartitions`|:white_check_mark:|
|`sparkSession`|:x:|
|`stat`|:white_check_mark:|
|`storageLevel`|:white_check_mark:|
|`subtract`|:white_check_mark:|
|`summary`|:white_check_mark:|
|`tail`|:white_check_mark:|
|`take`|:white_check_mark:|
|`to`|:x:|
|`toArrow`|:white_check_mark:|
|`toDF`|:x:|
|`toJSON`|:x:|
|`toLocalIterator`|:x:|
|`toPandas`|:heavy_minus_sign:|
|`transform`|:x:|
|`union`|:white_check_mark:|
|`unionAll`|:white_check_mark:|
|`unionByName`|:white_check_mark:|
|`unpersist`|:white_check_mark:|
|`unpivot`|:white_check_mark:|
|`where`|:white_check_mark:|
|`withColumn`|:white_check_mark:|
|`withColumnRenamed`|:white_check_mark:|
|`withColumns`|:white_check_mark:|
|`withColumnsRenamed`|:white_check_mark:|
|`withMetadata`|:white_check_mark:|
|`withWatermark`|:white_check_mark:|
|`write`|:white_check_mark: |
|`writeStream`|:x:|
|`writeTo`|:x:|
Contributor guide
Assessment
This issue has not been assessed yet.