oxidecomputer / oxidecomputer/dropshot

Make page tokens shorter

Open
#923 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
1.2k
Forks
104
Avg merge
1d 19h
Merged PRs (30d)
22

Description

This is helpful but not essential for https://github.com/oxidecomputer/console/issues/1102, putting page tokens in console URLs. The real blocker is #436.


Page tokens are base64ed JSON and they come out pretty long. Here's an example:

> atob('eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJuYW1lX2FzY2VuZGluZyIsInByb2plY3QiOiJhbGFuIiwibGFzdF9zZWVuIjoienp6LWluc3QtMTE0In19')
'{"v":"v1","page_start":{"sort_by":"name_ascending","project":"alan","last_seen":"zzz-inst-114"}}' 

oxiderack.com/projects/mock-project/instances?page=eyJ2IjoidjEiLCJwYWdlX3N0YXJ0Ijp7InNvcnRfYnkiOiJuYW1lX2FzY2VuZGluZyIsInByb2plY3QiOiJhbGFuIiwibGFzdF9zZWVuIjoienp6LWluc3QtMTE0In19 would work fine I guess, but to me it looks a bit silly. Here's where we do the serialization:

https://github.com/oxidecomputer/dropshot/blob/6b410861c105052faa30536edc9d271b2abf1be7/dropshot/src/pagination.rs#L417-L430

It should be pretty easy to encode the token in a more efficient format like MessagePack and maybe reduce the size of the data itself by, e.g., making some keys shorter.

Method Length of token
Current: base64 JSON string 128
MessagePack JSON as-is 104
MessagePack JSON with page_start -> p 92
MessagePack struct directly with serde_rmp (code below) 52
Rust program to compare base64 JSON and MessagePack
# Cargo.toml
[package]
name = "serialization-test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
serde = { version = "1.0", features = ["derive"] }
serde_json = "1.0"
rmp-serde = "0.15.4"
base64 = "0.13.0"
// src/main.rs

extern crate base64;
extern crate rmp_serde as rmps;
extern crate serde;
extern crate serde_json;

use serde::{Deserialize, Serialize};
use std::error::Error;

#[derive(Debug, PartialEq, Deserialize, Serialize)]
struct MyStruct {
    v: String,
    page_start: PageStart,
}

#[derive(Debug, PartialEq, Deserialize, Serialize)]
struct PageStart {
    sort_by: String,
    project: String,
    last_seen: String,
}

fn main() -> Result<(), Box<dyn Error>> {
    let data = MyStruct {
        v: "v1".to_owned(),
        page_start: PageStart {
            sort_by: "name_ascending".to_owned(),
            project: "alan".to_owned(),
            last_seen: "zzz-inst-114".to_owned(),
        },
    };

    // Serialize with JSON
    let json_data = serde_json::to_vec(&data)?;
    let encoded_json = base64::encode(&json_data);

    // Serialize with MessagePack
    let msgpack_data = rmps::to_vec(&data)?;
    let encoded_msgpack = base64::encode(&msgpack_data);

    // Compare the lengths of the encoded strings
    println!("Base64 JSON Length: {}", encoded_json.len());
    println!("Base64 MessagePack Length: {}", encoded_msgpack.len());

    Ok(())
}

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dropshot/src/pagination.rs around lines 417-430, where page tokens are serialized, and review issue #436 because the description identifies it as the real blocker. Compare the current base64 JSON approach with the proposed MessagePack alternatives, then verify that pagination tokens remain usable while becoming shorter.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend-api-design
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.