Better serialization for `query.batch` requests
A pull request for this has already been merged.
- #15991 by @Rich-Harris — merged
- Dominant language
- JavaScript
- Stars
- 20.8k
- Forks
- 2.3k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 156
Description
Describe the problem
query.batch calls return identically structured data.
However, current implementation does not seem to put this into optimization.
For example, in this dummy remote function which returns data shaped like this:
{
"1111111111": "b455",
"2222222222": "4b6b",
"3333333333": "85f6",
"0000000000": "46b87d61",
"4444444444": "c9c03b888ac8"
}
export const getWeather = query.batch(pipe(number(), integer()), (cityIds) => {
const lookup = new Map(
cityIds.map((id) => [
id,
Object.fromEntries(
randomUUID()
.split('-')
.map((str, idx) => [idx.toString().repeat(10), str]),
),
]),
);
return (cityId) => lookup.get(cityId);
});
The keys are repeated in each results, increasing the size of the serialized data.
"1111111111", "2222222222", "3333333333", "0000000000", "4444444444"
const data = {
type: 'result',
result: `[[1,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40],{"type":2,"data":3},"result",
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"b455\\",\\"4b6b\\",\\"85f6\\",\\"46b87d61\\",\\"c9c03b888ac8\\"]",{"type":2,"data":5},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"0a47\\",\\"4acc\\",\\"b7b9\\",\\"7482f37d\\",\\"206f4669156f\\"]",{"type":2,"data":7},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"cb4e\\",\\"4d8c\\",\\"956e\\",\\"33b9e02d\\",\\"19f3078de817\\"]",{"type":2,"data":9},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"b3d6\\",\\"4a4c\\",\\"9016\\",\\"210c1b21\\",\\"e503cec7ef40\\"]",{"type":2,"data":11},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"5836\\",\\"41a6\\",\\"80b1\\",\\"a14ee305\\",\\"b5cbd01c81dc\\"]",{"type":2,"data":13},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"e86c\\",\\"4549\\",\\"8807\\",\\"3443727c\\",\\"f5fe0714ef55\\"]",{"type":2,"data":15},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"83a6\\",\\"4dcb\\",\\"bda0\\",\\"dafb3add\\",\\"5f4f38654f3f\\"]",{"type":2,"data":17},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"89cb\\",\\"41af\\",\\"864d\\",\\"b7a932b8\\",\\"aad2e1776ef0\\"]",{"type":2,"data":19},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"5b95\\",\\"45f4\\",\\"ba51\\",\\"16712f7f\\",\\"279642fb42b2\\"]",{"type":2,"data":21},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"8a47\\",\\"4bff\\",\\"ac9e\\",\\"a3e7ec8c\\",\\"d3b46ae4512f\\"]",{"type":2,"data":23},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"1182\\",\\"4a8b\\",\\"a4f6\\",\\"8be2d913\\",\\"419eef17decd\\"]",{"type":2,"data":25},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"f9d1\\",\\"4585\\",\\"a9fb\\",\\"f7d3b069\\",\\"5992526724bc\\"]",{"type":2,"data":27},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"bf30\\",\\"4128\\",\\"9ad7\\",\\"0fbb2e7a\\",\\"aa8af8d2aa4c\\"]",{"type":2,"data":29},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"4a54\\",\\"412f\\",\\"b14a\\",\\"74cc15ec\\",\\"dd1bd73040aa\\"]",{"type":2,"data":31},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"c6c0\\",\\"4b97\\",\\"b241\\",\\"76653bba\\",\\"ed1304851678\\"]",{"type":2,"data":33},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"18ab\\",\\"487d\\",\\"a8f5\\",\\"87a1853b\\",\\"ec871915aeb5\\"]",{"type":2,"data":35},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"0d47\\",\\"402e\\",\\"9b49\\",\\"4b777c03\\",\\"68908658b1f6\\"]",{"type":2,"data":37},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"502d\\",\\"47ea\\",\\"8faf\\",\\"7052756b\\",\\"5bf44a58f212\\"]",{"type":2,"data":39},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"58b5\\",\\"49cf\\",\\"b942\\",\\"cd68f655\\",\\"1e17d377268f\\"]",{"type":2,"data":41},
"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"172b\\",\\"41f3\\",\\"90b1\\",\\"d951f0d3\\",\\"a93b6c7e98ce\\"]"]`,
};
Describe the proposed solution
For independent query.batch calls, devalue might not be the best option?
I should probably return tuple as a workaround.
const data = {
type: 'result',
result: `[[1,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40],{"type":2,"data":3},"result",
"[[1,2,3,4,5],\\"7e2fc191\\",\\"4a99\\",\\"432f\\",\\"80de\\",\\"7c265b0461da\\"]",{"type":2,"data":5},
"[[1,2,3,4,5],\\"89937f49\\",\\"c3d8\\",\\"4823\\",\\"b255\\",\\"3ee5b2f82e39\\"]",{"type":2,"data":7},
"[[1,2,3,4,5],\\"576c4c0b\\",\\"156c\\",\\"4ed5\\",\\"95f0\\",\\"8b58f03c7686\\"]",{"type":2,"data":9},
"[[1,2,3,4,5],\\"47243e25\\",\\"8b9c\\",\\"4c67\\",\\"94ea\\",\\"c348ee74eeac\\"]"]`,
};
Alternatives considered
No response
Importance
would make my life easier
Additional Information
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the query.batch serialization path and inspect how devalue encodes repeated object keys. Compare the serialized output with the tuple workaround shown in the issue; done means independent batch results preserve their values while avoiding repeated keys. The issue names no file or test, so locating the relevant entry point is part of the work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- api, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100