[Functions] Upload nar file to start function task failed
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
**Describe the bug**
When we start task in separately pulsar-worker , upload nar file restful request failed.
**To Reproduce**
Steps to reproduce the behavior:
1. Run function workers separately
2. Submit function task to this worker and upload nar file as uploadedInputStream
3. Restful request blocked, and we got nothing
**Solve this issue**
We compare separately function-worker and function-worker start with broker.
Function-worker use independent PulsarWorkerService to run a worker except rest server.
When registering jersey Resource, it is slightly different.
WebService in Broker will resgiter JsonMapperProvider and MultiPartFeature every time in [addResourceServlet](https://github.com/apache/pulsar/blob/master/pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java) .
But in PulsarWorkerService there is only MultiPartFeature [org.apache.pulsar.functions.worker.rest.Resources.](https://github.com/apache/pulsar/blob/master/pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/Resources.java).
**Additional context**
We use this script to submit task. It worker when function worker start with broker.
When submit to function-worker, we got a 400 ERROR like this:
> 2022-06-19T16:01:51,280+0800 [function-web-24-1] INFO org.eclipse.jetty.server.RequestLog - 127.0.0.1 - - [19/Jun/2022:16:01:50 +0800] "POST /admin/v3/sinks/public/default/pulsar-io-test2/ HTTP/1.1" 400 549 "-" "python-requests/2.6.0 CPython/2.7.5 Linux/3.10.107-1-tlinux2_kvm_guest-0049" 688
`
import json
import requests
from requests_toolbelt import MultipartEncoder
config = {
"classname": "org.apache.pulsar.io.kafka.KafkaBytesSink",
"inputs": ["topic_name"],
"configs": {
"bootstrapServers": "localhost:9092",
"topic": "test",
"acks": "1",
"batchSize": "16384",
"maxRequestSize": "1048576",
"producerConfigProperties":
{
"client.id": "test-pulsar-producer",
"security.protocol": "SASL_PLAINTEXT",
"sasl.mechanism": "GSSAPI",
"sasl.kerberos.service.name": "kafka",
"acks": "all"
}
}
}
url = "http://localhost:8080/admin/v3/sinks/public/default/pulsar-io-test/"
mp_encoder = MultipartEncoder(
fields={"%sConfig" % "sink": (None, json.dumps(config), "application/json"),
"data": ('pulsar-io-kafka-2.10.0.nar',
open('/opt/apache-pulsar-2.10.0/pulsar-io-kafka-2.10.0.nar', 'rb'))
}
)
res = requests.post(url, data=mp_encoder, headers={"Content-Type": mp_encoder.content_type})
print(res.status_code, res.text)
`
Contributor guide
Research direction
Compare pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/Resources.java with pulsar-broker/src/main/java/org/apache/pulsar/broker/web/WebService.java, focusing on resource registration and multipart handling. Reproduce the separate-worker upload using the provided request script, then verify that submitting a NAR file no longer returns HTTP 400 or blocks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100