into(Spark/HDFS, SQL DBs)
- Dominant language
- Python
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
migration from blaze
https://github.com/ContinuumIO/blaze/issues/582
reposting for ease of use:
from @chdoig
As Spark is becoming a popular backend, it's a common use case for people to want to transfer their datasets in DBs to a Spark/HDFS cluster.
It would be nice to have an easy interface for end-users to transfer their tables in DBs to a Cluster.
into(Spark/HDFS, SQL DBs)
A lot of people are talking now about tachyon, maybe worth taking a look:
http://tachyon-project.org/
http://ampcamp.berkeley.edu/big-data-mini-course/tachyon.html
This might be related with @quasiben work on SparkSQL. Maybe a barrier for people to star using SparkSQL is how they should make that transfer since:
> A SchemaRDD can be created from an existing RDD, Parquet file, a JSON dataset, or by running HiveQL against data stored in Apache Hive.
But I'm not able to find how you make that connection from existing SQL DBs:
http://spark.apache.org/docs/latest/sql-programming-guide.html
cc: @mrocklin
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.