influxdata / influxdata/influxdb-client-js
Unable to write points for long run (unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824))
- Dominant language
- TypeScript
- Stars
- 352
- Forks
- 69
- Avg merge
- 10m
- Merged PRs (30d)
- 11
Description
### Specifications
* Client Version: "@influxdata/influxdb-client": "^1.33.2",
"@influxdata/influxdb-client-apis": "^1.33.2",
* InfluxDB Version: OSS v2.6
* Platform: Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz 2.59 GHz
64.0 GB (63.7 GB usable)
64-bit operating system, x64-based processor
### Code sample to reproduce problem
Using Singleton InfluxDB client instance used for all writes & Single writeApi used for all calls throughout
Call details: 200 parallel calls with each call having 500 samples/points, this is equal to 1lakh measurements (or point data).
These 200 calls are run at every 1 second interval.
Single call payload size in bytes: 85452 and 200 calls its ~16MB
```javascript
constructor code:
private writeOptions: any = {
batchSize: 1000,
flushInterval: 500,
// maxBatchBytes: 1024*1024*10,
writeFailed: this.writeFailedHandler,
// // function (this: WriteApi, error: Error, lines: string[], attempt: number, expires: number): void | Promise {
// // throw new Error('Function not implemented.');
// // },
writeSuccess: this.writeSuccessHandler,
// // function (this: WriteApi, lines: string[]): void {
// // throw new Error('Function not implemented.');
// // },
// writeRetrySkipped: function (entry: { lines: string[]; expires: number; }): void {
// throw new Error('Function not implemented.');
// },
// maxRetries: 0,
// maxRetryTime: 0,
maxBufferLines: 0,
// retryJitter: 0,
// minRetryDelay: 0,
// maxRetryDelay: 0,
// exponentialBase: 0,
// randomRetry: false
};
this.influxClient = new InfluxDB({ url: envCfg.url, token: envCfg.token });
this.writeApi = this.influxClient.getWriteApi(envCfg.org, envCfg.bucket, 'ns', this.writeOptions);
////// method call as per above setup i.e. 200 parallel calls every second. total points in 1 call = 500
public async setData(setDataReq: SetDataRequest) {
let dataPoints: Point[] = [];
setDataReq.SampleSets.forEach(sampleSet => {
for (let sampleIndex = 0; sampleIndex < sampleSet.Samples.length; sampleIndex++) {
const sample = sampleSet.Samples[sampleIndex];
let point = new Point(sampleSet.TagId)
.intField('hashId', stringHash(sampleSet.TagId))
.floatField('value', sample.Value)
.uintField('dataStatus', sample.DataStatus)
.uintField('nodeStatus', sample.NodeStatus)
.booleanField('isValid', true)
.timestamp(new Date(convertUtcTicksToLocalDate(sample.Timestamp).toJSON()));
dataPoints.push(point);
}
});
const st = performance.now();
this.writeApi.writePoints(dataPoints);
if ((this.logCounter++) != 0 && (this.logCounter % this.logAt) == 0) {
console.log(`SetData for samples:ellapedInMs ${dataPoints.length}:${(performance.now() - st)}`);
this.logCounter = 0;
}
}
```
Note that i'm not closing/reset write api any time, as I expect to the writeOptions should do the job for me for long running write data.
### Expected behavior
Should be able to write data seemlessly without failures for long run time like many days.
Memory should be efficiently managed by client-js & influxd by itself.
Kindly provide best way to acheive such write load without having to close/reset/flush the writeApi.
Thanks & Regards!
### Actual behavior
node js influx client: failed at ~190 times, when batchsize = 5000 & flush=500ms
i.e failed in 3/5 mins
--
ERROR: RetryBuffer: 15000 oldest lines removed to keep buffer size under the limit of 32000 lines.
Error in write HttpError: unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)
WARN: Write to InfluxDB failed (attempt: 1). m [HttpError]: unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)
at IncomingMessage. (C:\BH-Root\Code\POC\s1e-poc-timeseries\timeseries-js\node_modules\@influxdata\influxdb-client\dist\index.js:5:5671)
at IncomingMessage.emit (node:events:402:35)
at endReadableNT (node:internal/streams/readable:1343:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
statusCode: 500,
statusMessage: 'Internal Server Error',
body: '{"code":"internal error","message":"unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)"}',
contentType: 'application/json; charset=utf-8',
json: {
code: 'internal error',
message: 'unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)'
},
code: 'internal error',
_retryAfter: 0
}
single payload size in bytes: 85452
Failed: with batchsize = 1000 & flush = 500ms. i.e. failed in ~20 mins
{"level":"error","message":"3/8/2023, 3:33:42 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"error","message":"3/8/2023, 3:33:42 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"info","message":"timesCalled 1186"}
Failure 2:
{"level":"info","message":"timesCalled 946"}
{"level":"error","message":"3/8/2023, 3:28:36 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"error","message":"3/8/2023, 3:28:36 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"info","message":"3/8/2023, 3:28:36 PM Req size= 85475 B. Res: {\"data\":\"resData\"}"}
{"level":"info","message":"timesCalled 947"}
Error in write Error: read ECONNRESET
WARN: Write to InfluxDB failed (attempt: 1). Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:220:20) {
errno: -4077,
code: 'ECONNRESET',
syscall: 'read'
}

### Additional info
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the getWriteApi and writePoints entry points described in issue #739, then inspect the client code corresponding to node_modules/@influxdata/influxdb-client/dist/index.js and its retry-buffer handling. Reproduce the reported sustained write load with the supplied batch and flush settings. Done should clearly establish the client-side behavior and include a tested resolution or documented limitation for long-running writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100