Error 500 on heavy load.

I have experienced a strange behaviour over the REST interface using node-fetch module.

While this is not the most efficient way of doing things, it has thrown up the following error 500.

I read in around 11 ttl files so that total about 1,6m triples. I now want to pivot them and create a file per subject. I process each of the 1,6m triples as set out below. Check whether the resource exists. If it does not, create it using PUT, else PATCH in the new triple.

For logging I write out a 'pulse' every 5,000 records.

The code below is an extract of the process.

   let getResult = await fetch(newUrl, {method: "HEAD"})
//                console.log(getResult.status)
                if (getResult.status === 404) {
                    options = {
                        method: "PUT",
                        headers: {
                            "content-type" : "text/turtle",
                            "Link": "<http://www.w3.org/ns/ldp#Resource>; rel='type'"
                        },
                        body: `${newSubject} <${newPredicate}> ${newObject} .`
                   }
                }
                else {
                    options = {
                        method: "PATCH",
                        headers: {
                            "content-type" : "application/sparql-update",
                            "Link": "<http://www.w3.org/ns/ldp#Resource>; rel='type'"
                        },
                        body: `INSERT DATA { ${newSubject} <${newPredicate}> ${newObject} }`
                    }
                };
                let res1 = await fetch(newUrl, options);
                if ((res1.status != 200) && (res1.status != 201)) console.log("PROBLEMS WITH THE SOLID-SERVER", res1.status, JSON.stringify(options));
                if ( i % 5000 === 0) console.log("output", i, newUrl, res1.status);
 

Just after the 80,000th log entry and before the 85,000th log entry, the server starts producing error 500 and never recovers. As can be seen in the output below:

...
output 75000 https://127.0.0.1:8443/eim/App_Inst/714101 200
output 80000 https://127.0.0.1:8443/eim/App_Inst/876505 200
PROBLEMS WITH THE SOLID-SERVER 500 {"method":"PUT","headers":{"content-type":"text/turtle","Link":"<http://www.w3.org/ns/ldp#Resource>; rel='type'"},"body":"<> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://127.0.0.1:8443/eim/App_Instance_Review> ."}
PROBLEMS WITH THE SOLID-SERVER 500 {"method":"PUT","headers":{"content-type":"text/turtle","Link":"<http://www.w3.org/ns/ldp#Resource>; rel='type'"},"body":"<> <https://127.0.0.1:8443/eim/App_Inst_ID> \"\" ."}
...

Inizia dall’interfaccia REST e riproduci il carico di lavoro di node-fetch che elabora circa 1,6 milioni di triple, utilizzando le richieste PUT e PATCH riportate. Indaga sul motivo per cui il server inizia a restituire risposte HTTP 500 dopo circa 80.000 record e verifica se si riprende; il lavoro è considerato completato quando il carico di lavoro non produce più errori 500 persistenti e il problema è coperto da un test o da una riproduzione appropriata.