microsoft / microsoft/SynapseML
Make Spark Serving Easier to Use
@mhamilton723 is already working on this.
Since Jul 18, 2018.
- Dominant language
- Scala
- Stars
- 5.2k
- Forks
- 868
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 45
Description
- [ ] Add option to fail when it cannot parse the input
- [x] Add tests on ADB
- [x] Add syntactic sugars for adding input parsers
- [x] Add ability to reply with other types
- [ ] Fix offsets commited out of order exception
Repro:
```python
import mmlspark
from pyspark.sql.functions import length, col, lit, from_json
from pyspark.sql.types import *
df = spark.readStream.server() \
.address("localhost", 8888, "my_api") \
.load()
server = df.withColumn("reply", length(col("value"))) \
.withColumn("replyString", col("reply").cast(StringType()))\
.writeStream \
.server() \
.option("name", "my_api") \
.queryName("my_query") \
.option("replyCol", "replyString") \
.option("checkpointLocation", "checkpoints") \
.start()
import requests
r = requests.post(data="{foo:1}", url="http://localhost:8888/my_api")
print(r.text)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.