Error 500 on heavy load.

I have experienced a strange behaviour over the REST interface using node-fetch module.

While this is not the most efficient way of doing things, it has thrown up the following error 500.

I read in around 11 ttl files so that total about 1,6m triples. I now want to pivot them and create a file per subject. I process each of the 1,6m triples as set out below. Check whether the resource exists. If it does not, create it using PUT, else PATCH in the new triple.

For logging I write out a 'pulse' every 5,000 records.

The code below is an extract of the process.

   let getResult = await fetch(newUrl, {method: "HEAD"})
//                console.log(getResult.status)
                if (getResult.status === 404) {
                    options = {
                        method: "PUT",
                        headers: {
                            "content-type" : "text/turtle",
                            "Link": "<http://www.w3.org/ns/ldp#Resource>; rel='type'"
                        },
                        body: `${newSubject} <${newPredicate}> ${newObject} .`
                   }
                }
                else {
                    options = {
                        method: "PATCH",
                        headers: {
                            "content-type" : "application/sparql-update",
                            "Link": "<http://www.w3.org/ns/ldp#Resource>; rel='type'"
                        },
                        body: `INSERT DATA { ${newSubject} <${newPredicate}> ${newObject} }`
                    }
                };
                let res1 = await fetch(newUrl, options);
                if ((res1.status != 200) && (res1.status != 201)) console.log("PROBLEMS WITH THE SOLID-SERVER", res1.status, JSON.stringify(options));
                if ( i % 5000 === 0) console.log("output", i, newUrl, res1.status);
 

Just after the 80,000th log entry and before the 85,000th log entry, the server starts producing error 500 and never recovers. As can be seen in the output below:

...
output 75000 https://127.0.0.1:8443/eim/App_Inst/714101 200
output 80000 https://127.0.0.1:8443/eim/App_Inst/876505 200
PROBLEMS WITH THE SOLID-SERVER 500 {"method":"PUT","headers":{"content-type":"text/turtle","Link":"<http://www.w3.org/ns/ldp#Resource>; rel='type'"},"body":"<> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <https://127.0.0.1:8443/eim/App_Instance_Review> ."}
PROBLEMS WITH THE SOLID-SERVER 500 {"method":"PUT","headers":{"content-type":"text/turtle","Link":"<http://www.w3.org/ns/ldp#Resource>; rel='type'"},"body":"<> <https://127.0.0.1:8443/eim/App_Inst_ID> \"\" ."}
...

Commencez par l’interface REST et reproduisez la charge de travail de node-fetch qui traite environ 1,6 million de triplets, en utilisant les requêtes PUT et PATCH signalées. Examinez pourquoi le serveur commence à renvoyer des réponses HTTP 500 après environ 80 000 enregistrements et s’il récupère ; le travail est considéré comme terminé lorsque la charge de travail ne produit plus d’erreurs 500 persistantes et que l’échec est couvert par un test ou une reproduction approprié(e).