Azure / Azure/azure-functions-durable-js

Investigate possible scaling issues when there are lots of outbound HTTP calls

Open
#158 9 comments 0 reactions 0 assignees View on GitHub
bug P2
Dominant language
TypeScript
Stars
142
Forks
66
Avg merge
3d 19h
Merged PRs (30d)
4

Description

Now that #152 is completed, we should rerun some scale tests to see if things have improved.

Pasting in some findings from tests I ran in late December:

---

Did a bit more testing on durable scaling using @brandonh-msft's [sample app](https://github.com/brandonh-msft/durablefunctions-javascript-scaletest). Ran into a few issues (some old, some new) but I think I have a better idea of what's happening now. Sharing some initial findings:

TL;DR: I managed to get it working by making these changes:
- Change from request to Axios as the HTTP client
- Limit the `maxSockets` on the HTTPS agent used by Axios
- Rate limiting the number of outgoing requests per instance using [Bottleneck](https://www.npmjs.com/package/bottleneck)

The code for this is here: https://github.com/anthonychu/durablefunctions-javascript-scaletest/tree/20191229-refactor/javascript-axios

The main issue appears to be port exhaustion. There are 2 main types of errors: `EADDRINUSE` and `ETIMEDOUT`. By limiting the `maxSockets` on the HTTPS agent, it seemed to have eliminated the `EADDRINUSE` issue.

However, there are still occasional `ETIMEDOUT` errors. They appear with the IP address of the API that the activity function is calling. I don't think the remote API that it is calling is down or timing out; I think it's a problem with the app itself. (Also see `ETIMEDOUT` error below when the client calls the host)

Using Bottleneck to limit outbound API calls to 4 per second appears to eliminate these `ETIMEDOUT` errors. I haven't tried increasing that number.

I've also tried a couple of other versions of the code that didn't work:
- Modified Brandon's original app to use Bottleneck to rate limit outbound calls ([code](https://github.com/anthonychu/durablefunctions-javascript-scaletest/tree/20191229-refactor/javascript-request-bottleneck))
- Created a version that uses callHttp() instead of an activity function ([code](https://github.com/anthonychu/durablefunctions-javascript-scaletest/tree/20191229-refactor/javascript-durablehttp))

Both of these encountered a different `ETIMEDOUT` error. The calls between the JS worker and the host via the frontend were failing when trying to invoke `DurableOrchestrationClient.startNew()`.

```json
{
"message": "connect ETIMEDOUT 40.112.243.4:443",
"name": "Error",
"stack": "Error: connect ETIMEDOUT 40.112.243.4:443\n at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1106:14)",
"config": {
"url": "https://test-javascript-scaletest-request-20191229.azurewebsites.net/runtime/webhooks/durabletask/orchestrators/StartWorkflow?code=***********************************************",
"method": "post",
"data": "",
"headers": {
"Accept": "application/json, text/plain, */*",
"Content-Type": "application/json",
"User-Agent": "axios/0.19.0"
},
"transformRequest": [
null
],
"transformResponse": [
null
],
"timeout": 0,
"xsrfCookieName": "XSRF-TOKEN",
"xsrfHeaderName": "X-XSRF-TOKEN",
"maxContentLength": null
},
"code": "ETIMEDOUT"
}
```

I'm not sure why this error didn't happen in the Axios implementation. I wonder if limiting the `maxSockets` and other settings in Axios was also affecting the instance of Axios used in the Durable Functions JS SDK.

Contributor guide

Open the contributing guide

Research direction

Start by rerunning the scale tests from the linked durablefunctions-javascript-scaletest sample, including the Axios and Durable HTTP variants, after reviewing the changes from #152. Compare the EADDRINUSE and ETIMEDOUT failures under socket limits and outbound rate limiting; done means documenting whether the issue persists, its reproducible conditions, and the next change needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, typescript
Domain
backend, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.