sidorares / sidorares/node-mysql2

Should not reach here: undefined

Open
#3,217 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

needs reproduction
Dominant language
TypeScript
Stars
4.4k
Forks
680
Avg merge
9h 7m
Merged PRs (30d)
59

Description

I am using stream to move a big data set, but it seems there is a cap of ~460 iterations that causes an error i dont understand.

This is my code:

let i = 0;
const stream = connection.query('SELECT * FROM schedules ORDER BY id ASC').stream();
stream.on('data', result => {
	stream.pause();
	db.collection('packages').deleteOne({ id: result.id }).then(() => {
		db.collection('packages').insertOne(result).then(() => {
			i++;
			if (i % 1000 === 0) {
				console.log('Processed', i);
			}
			stream.resume();
		});
	});
});
stream.on('end', () => {
	connection.end();
	mongoClient.close();
	console.log('Finished streaming', i);
});
      at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:231:13)
      at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
      at next (32:68)
      at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
      at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
      at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
      at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
      at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
      at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
227 |       res = resNumber.toString() === resString ? resNumber : resString;
228 |       return bigNumberStrings ? resString : res;
229 |     }
230 |     // eslint-disable-next-line no-console
231 |     console.trace();
232 |     throw new Error(`Should not reach here: ${tag}`);
                ^
error: Should not reach here: undefined
      at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:232:11)
      at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
      at next (32:68)
      at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
      at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
      at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
      at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
      at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
      at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
      at emit (node:events:183:48)

if i use async/await instead of thenables i have the same result, only when i dont do both it runs through, but then 3mio db operations stuck up in memory and make dev ops kill it.

all i want is to do this in the background a little slower.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the streaming query with the example and inspect node_modules/mysql2/lib/packets/packet.js, especially readLengthCodedNumberExt, along with lib/commands/query.js and lib/packet_parser.js. Trace the malformed or unexpected packet after repeated pause/resume calls; done means the failure is explained and streaming completes without accumulating all database operations in memory.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, mysql, nodejs
Domain
backend, database
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.