eclipse-thingweb / eclipse-thingweb/node-wot

Consume and read loop causes memory leak

Open
#1,062 8 comments 0 reactions 0 assignees View on GitHub
AJV core
Dominant language
TypeScript
Stars
192
Forks
100
Avg merge
3d 2h
Merged PRs (30d)
6

Description

I discovered this memory leak a week ago, but only today I finally got what is the issue. Take this simple code:
```js
const { Servient } = require("@node-wot/core");
const { HttpClientFactory } = require("@node-wot/binding-http");
const { Helpers } = require("@node-wot/core");

let WoTHelpers;
let api;
async function main(){
const servient = new Servient();
servient.addClientFactory(new HttpClientFactory());
api = await servient.start();
WoTHelpers = new Helpers(servient);
readloop();
}

async function readloop() {
const td = await WoTHelpers.fetch("http://localhost:8080/test")
const testThing = await api.consume(td);
console.log("Consumed");

const output = await testThing.readProperty("temperature")
const value = await output.value();

console.log(value);
setTimeout(() => {
main();
}, 500);
}

main();
```

As you can see, we are simply polling a remote web thing and reading its `temperature` property. The loop seems pretty harmless but as time goes by the heap starts to inflate and after some days your service will crash for not having free available memory.

Putting aside the reason why we are continuously fetching the consumed thing (more on that later), the root cause of this memory leak is `ajv` cache. Basically, every time we call the `value` function we call `ajv.compile(schema)` which, as a side effect, stores the result of the compilation inside an internal `Map` cache. The schema object is used as a key of this `Map` hence every time we fetch and consume the Thing Description we are creating a new schema object which for the map is a completely new key. This also triggers another compilation round which has an impact also in the CPU performance.

One easy solution for our users is simply to always avoid fetch-consume loops (btw the same applies to simply consuming the same Thing Description in a loop because we are cloning it every time). However, in my use case, we were continuously fetching to have some small tolerance in transient services shutdowns ( e.g. if the remote thing updates the consumer service recovers automatically ) .

One solution could be to handle the caching of the Thing Description schemas ourselves (`ajv` provides a good API for managing schemas manually), but this would require some major efforts since, in the current architecture, the `InteractionOutput` has the responsibility of handling schema validation and it has a narrow view of the whole ConsumeThing. We would need to move the compilation of schemas responsibility up to the ConsumedThing object and pass it as a parameter to the `InteractionOutput` object which only calls the validate function.

Any thoughts?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.