sveltejs / sveltejs/kit

Better serialization for `query.batch` requests

Open
#15,784 4 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #15991 by @Rich-Harris — merged
Dominant language
JavaScript
Stars
20.8k
Forks
2.3k
Avg merge
1d 16h
Merged PRs (30d)
156

Description

Describe the problem

query.batch calls return identically structured data.

However, current implementation does not seem to put this into optimization.

For example, in this dummy remote function which returns data shaped like this:

{
  "1111111111": "b455",
  "2222222222": "4b6b",
  "3333333333": "85f6",
  "0000000000": "46b87d61",
  "4444444444": "c9c03b888ac8"
}
export const getWeather = query.batch(pipe(number(), integer()), (cityIds) => {
  const lookup = new Map(
    cityIds.map((id) => [
      id,
      Object.fromEntries(
        randomUUID()
          .split('-')
          .map((str, idx) => [idx.toString().repeat(10), str]),
      ),
    ]),
  );
  return (cityId) => lookup.get(cityId);
});

The keys are repeated in each results, increasing the size of the serialized data.

"1111111111", "2222222222", "3333333333", "0000000000", "4444444444"
const data = {
	type: 'result',
	result: `[[1,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40],{"type":2,"data":3},"result",
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"b455\\",\\"4b6b\\",\\"85f6\\",\\"46b87d61\\",\\"c9c03b888ac8\\"]",{"type":2,"data":5},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"0a47\\",\\"4acc\\",\\"b7b9\\",\\"7482f37d\\",\\"206f4669156f\\"]",{"type":2,"data":7},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"cb4e\\",\\"4d8c\\",\\"956e\\",\\"33b9e02d\\",\\"19f3078de817\\"]",{"type":2,"data":9},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"b3d6\\",\\"4a4c\\",\\"9016\\",\\"210c1b21\\",\\"e503cec7ef40\\"]",{"type":2,"data":11},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"5836\\",\\"41a6\\",\\"80b1\\",\\"a14ee305\\",\\"b5cbd01c81dc\\"]",{"type":2,"data":13},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"e86c\\",\\"4549\\",\\"8807\\",\\"3443727c\\",\\"f5fe0714ef55\\"]",{"type":2,"data":15},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"83a6\\",\\"4dcb\\",\\"bda0\\",\\"dafb3add\\",\\"5f4f38654f3f\\"]",{"type":2,"data":17},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"89cb\\",\\"41af\\",\\"864d\\",\\"b7a932b8\\",\\"aad2e1776ef0\\"]",{"type":2,"data":19},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"5b95\\",\\"45f4\\",\\"ba51\\",\\"16712f7f\\",\\"279642fb42b2\\"]",{"type":2,"data":21},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"8a47\\",\\"4bff\\",\\"ac9e\\",\\"a3e7ec8c\\",\\"d3b46ae4512f\\"]",{"type":2,"data":23},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"1182\\",\\"4a8b\\",\\"a4f6\\",\\"8be2d913\\",\\"419eef17decd\\"]",{"type":2,"data":25},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"f9d1\\",\\"4585\\",\\"a9fb\\",\\"f7d3b069\\",\\"5992526724bc\\"]",{"type":2,"data":27},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"bf30\\",\\"4128\\",\\"9ad7\\",\\"0fbb2e7a\\",\\"aa8af8d2aa4c\\"]",{"type":2,"data":29},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"4a54\\",\\"412f\\",\\"b14a\\",\\"74cc15ec\\",\\"dd1bd73040aa\\"]",{"type":2,"data":31},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"c6c0\\",\\"4b97\\",\\"b241\\",\\"76653bba\\",\\"ed1304851678\\"]",{"type":2,"data":33},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"18ab\\",\\"487d\\",\\"a8f5\\",\\"87a1853b\\",\\"ec871915aeb5\\"]",{"type":2,"data":35},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"0d47\\",\\"402e\\",\\"9b49\\",\\"4b777c03\\",\\"68908658b1f6\\"]",{"type":2,"data":37},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"502d\\",\\"47ea\\",\\"8faf\\",\\"7052756b\\",\\"5bf44a58f212\\"]",{"type":2,"data":39},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"58b5\\",\\"49cf\\",\\"b942\\",\\"cd68f655\\",\\"1e17d377268f\\"]",{"type":2,"data":41},
	"[{\\"1111111111\\":1,\\"2222222222\\":2,\\"3333333333\\":3,\\"0000000000\\":4,\\"4444444444\\":5},\\"172b\\",\\"41f3\\",\\"90b1\\",\\"d951f0d3\\",\\"a93b6c7e98ce\\"]"]`,
};
Describe the proposed solution

For independent query.batch calls, devalue might not be the best option?

I should probably return tuple as a workaround.

const data = {
	type: 'result',
	result: `[[1,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40],{"type":2,"data":3},"result",
  "[[1,2,3,4,5],\\"7e2fc191\\",\\"4a99\\",\\"432f\\",\\"80de\\",\\"7c265b0461da\\"]",{"type":2,"data":5},
  "[[1,2,3,4,5],\\"89937f49\\",\\"c3d8\\",\\"4823\\",\\"b255\\",\\"3ee5b2f82e39\\"]",{"type":2,"data":7},
  "[[1,2,3,4,5],\\"576c4c0b\\",\\"156c\\",\\"4ed5\\",\\"95f0\\",\\"8b58f03c7686\\"]",{"type":2,"data":9},
  "[[1,2,3,4,5],\\"47243e25\\",\\"8b9c\\",\\"4c67\\",\\"94ea\\",\\"c348ee74eeac\\"]"]`,
};

Alternatives considered

No response

Importance

would make my life easier

Additional Information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the query.batch serialization path and inspect how devalue encodes repeated object keys. Compare the serialized output with the tuple workaround shown in the issue; done means independent batch results preserve their values while avoiding repeated keys. The issue names no file or test, so locating the relevant entry point is part of the work.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
api, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.