influxdata / influxdata/influxdb-client-js

Unable to write points for long run (unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824))

Open
#739 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
352
Forks
69
Avg merge
10m
Merged PRs (30d)
11

Description

### Specifications

* Client Version: "@influxdata/influxdb-client": "^1.33.2",
"@influxdata/influxdb-client-apis": "^1.33.2",
* InfluxDB Version: OSS v2.6
* Platform: Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz   2.59 GHz
64.0 GB (63.7 GB usable)
64-bit operating system, x64-based processor

### Code sample to reproduce problem

Using Singleton InfluxDB client instance used for all writes & Single writeApi used for all calls throughout
Call details: 200 parallel calls with each call having 500 samples/points, this is equal to 1lakh measurements (or point data).
These 200 calls are run at every 1 second interval.
Single call payload size in bytes: 85452 and 200 calls its ~16MB

```javascript

constructor code:
private writeOptions: any = {
batchSize: 1000,
flushInterval: 500,
// maxBatchBytes: 1024*1024*10,

writeFailed: this.writeFailedHandler,
// // function (this: WriteApi, error: Error, lines: string[], attempt: number, expires: number): void | Promise {
// // throw new Error('Function not implemented.');
// // },
writeSuccess: this.writeSuccessHandler,
// // function (this: WriteApi, lines: string[]): void {
// // throw new Error('Function not implemented.');
// // },
// writeRetrySkipped: function (entry: { lines: string[]; expires: number; }): void {
// throw new Error('Function not implemented.');
// },
// maxRetries: 0,
// maxRetryTime: 0,
maxBufferLines: 0,
// retryJitter: 0,
// minRetryDelay: 0,
// maxRetryDelay: 0,
// exponentialBase: 0,
// randomRetry: false
};
this.influxClient = new InfluxDB({ url: envCfg.url, token: envCfg.token });
this.writeApi = this.influxClient.getWriteApi(envCfg.org, envCfg.bucket, 'ns', this.writeOptions);

////// method call as per above setup i.e. 200 parallel calls every second. total points in 1 call = 500

public async setData(setDataReq: SetDataRequest) {
let dataPoints: Point[] = [];

setDataReq.SampleSets.forEach(sampleSet => {
for (let sampleIndex = 0; sampleIndex < sampleSet.Samples.length; sampleIndex++) {
const sample = sampleSet.Samples[sampleIndex];

let point = new Point(sampleSet.TagId)
.intField('hashId', stringHash(sampleSet.TagId))
.floatField('value', sample.Value)
.uintField('dataStatus', sample.DataStatus)
.uintField('nodeStatus', sample.NodeStatus)
.booleanField('isValid', true)
.timestamp(new Date(convertUtcTicksToLocalDate(sample.Timestamp).toJSON()));
dataPoints.push(point);
}
});
const st = performance.now();
this.writeApi.writePoints(dataPoints);
if ((this.logCounter++) != 0 && (this.logCounter % this.logAt) == 0) {
console.log(`SetData for samples:ellapedInMs ${dataPoints.length}:${(performance.now() - st)}`);
this.logCounter = 0;
}
}

```
Note that i'm not closing/reset write api any time, as I expect to the writeOptions should do the job for me for long running write data.

### Expected behavior

Should be able to write data seemlessly without failures for long run time like many days.
Memory should be efficiently managed by client-js & influxd by itself.
Kindly provide best way to acheive such write load without having to close/reset/flush the writeApi.
Thanks & Regards!

### Actual behavior

node js influx client: failed at ~190 times, when batchsize = 5000 & flush=500ms
i.e failed in 3/5 mins
--
ERROR: RetryBuffer: 15000 oldest lines removed to keep buffer size under the limit of 32000 lines.
Error in write HttpError: unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)
WARN: Write to InfluxDB failed (attempt: 1). m [HttpError]: unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)
at IncomingMessage. (C:\BH-Root\Code\POC\s1e-poc-timeseries\timeseries-js\node_modules\@influxdata\influxdb-client\dist\index.js:5:5671)
at IncomingMessage.emit (node:events:402:35)
at endReadableNT (node:internal/streams/readable:1343:12)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
statusCode: 500,
statusMessage: 'Internal Server Error',
body: '{"code":"internal error","message":"unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)"}',
contentType: 'application/json; charset=utf-8',
json: {
code: 'internal error',
message: 'unexpected error writing points to database: engine: cache-max-memory-size exceeded: (1074087500/1073741824)'
},
code: 'internal error',
_retryAfter: 0
}

single payload size in bytes: 85452
 

Failed: with batchsize = 1000 & flush = 500ms. i.e. failed in ~20 mins
{"level":"error","message":"3/8/2023, 3:33:42 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"error","message":"3/8/2023, 3:33:42 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"info","message":"timesCalled 1186"}

Failure 2:
{"level":"info","message":"timesCalled 946"}
{"level":"error","message":"3/8/2023, 3:28:36 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"error","message":"3/8/2023, 3:28:36 PM, Error: connect ECONNREFUSED 127.0.0.1:7000"}
{"level":"info","message":"3/8/2023, 3:28:36 PM Req size= 85475 B. Res: {\"data\":\"resData\"}"}
{"level":"info","message":"timesCalled 947"}
Error in write Error: read ECONNRESET
WARN: Write to InfluxDB failed (attempt: 1). Error: read ECONNRESET
at TCP.onStreamRead (node:internal/stream_base_commons:220:20) {
errno: -4077,
code: 'ECONNRESET',
syscall: 'read'
}
 
![image](https://user-images.githubusercontent.com/88302279/223696767-af781110-71bf-420c-93d8-8fb4f1862993.png)

### Additional info

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the getWriteApi and writePoints entry points described in issue #739, then inspect the client code corresponding to node_modules/@influxdata/influxdb-client/dist/index.js and its retry-buffer handling. Reproduce the reported sustained write load with the supplied batch and flush settings. Done should clearly establish the client-side behavior and include a tested resolution or documented limitation for long-running writes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.