sidorares / sidorares/node-mysql2
Should not reach here: undefined
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 4.4k
- Forks
- 680
- Avg merge
- 9h 7m
- Merged PRs (30d)
- 59
Description
I am using stream to move a big data set, but it seems there is a cap of ~460 iterations that causes an error i dont understand.
This is my code:
let i = 0;
const stream = connection.query('SELECT * FROM schedules ORDER BY id ASC').stream();
stream.on('data', result => {
stream.pause();
db.collection('packages').deleteOne({ id: result.id }).then(() => {
db.collection('packages').insertOne(result).then(() => {
i++;
if (i % 1000 === 0) {
console.log('Processed', i);
}
stream.resume();
});
});
});
stream.on('end', () => {
connection.end();
mongoClient.close();
console.log('Finished streaming', i);
});
at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:231:13)
at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
at next (32:68)
at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
227 | res = resNumber.toString() === resString ? resNumber : resString;
228 | return bigNumberStrings ? resString : res;
229 | }
230 | // eslint-disable-next-line no-console
231 | console.trace();
232 | throw new Error(`Should not reach here: ${tag}`);
^
error: Should not reach here: undefined
at readLengthCodedNumberExt (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:232:11)
at readLengthCodedString (/home/bun/app/node_modules/mysql2/lib/packets/packet.js:381:22)
at next (32:68)
at row (/home/bun/app/node_modules/mysql2/lib/commands/query.js:242:29)
at execute (/home/bun/app/node_modules/mysql2/lib/commands/command.js:45:22)
at handlePacket (/home/bun/app/node_modules/mysql2/lib/connection.js:481:34)
at /home/bun/app/node_modules/mysql2/lib/connection.js:97:12
at executePayload (/home/bun/app/node_modules/mysql2/lib/packet_parser.js:139:14)
at /home/bun/app/node_modules/mysql2/lib/connection.js:104:25
at emit (node:events:183:48)
if i use async/await instead of thenables i have the same result, only when i dont do both it runs through, but then 3mio db operations stuck up in memory and make dev ops kill it.
all i want is to do this in the background a little slower.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the streaming query with the example and inspect node_modules/mysql2/lib/packets/packet.js, especially readLengthCodedNumberExt, along with lib/commands/query.js and lib/packet_parser.js. Trace the malformed or unexpected packet after repeated pause/resume calls; done means the failure is explained and streaming completes without accumulating all database operations in memory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, mysql, nodejs
- Domain
- backend, database
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100