oxidecomputer / oxidecomputer/serde_tokenstream

Serde -> TokenStream serializer

Open
#220 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
77
Forks
3
Avg merge
2d 12h
Merged PRs (30d)
11

Description

Hi!

Have you considered implementing a reverse operation: serialization to TokenStream as part of this library?

Having a round-trip operation could be very useful in some contexts: one is a wrapper around macros using deserialization with serde_tokenstream. Because Rust macros are expanded recursively, one could write a macro such as:

#[simpler_api(param = "value")]
struct S(...);

that could expand to:

#[complex_api(
    param1 = 42,
    param2 = ["v", "a", "l", "u", "e"],
)]
struct S(...);

Of course, generating the parameters could be done manually, but having a complete round-trip operation as part of a single library could enable complex macro shenanigans. For instance, the macro_magic crate enables exporting tokens at one location, and then receiving them by a proc macro at completely different place (even across file and crate boundary). This is done with generation of decl macros doing callbacks to proc macros with a token tree. Combining this idea with de/serialization to valid token trees, we could have macros emit data-objects to be consumed by other proc-macros, effectively enabling sound communication across proc-macros invocation.

Collapsed: example data-macros expansion.

This is how one may implement data passing across various proc macros by using serialization to a TokenStream. In this example we do just a simple analysis that could be done using existing macro_magic: dump_analysis proc macro generates some data that is consumed by another proc-macro handler.

// A call such as
#[dump_analysis]
pub struct ApiCall {
    field: u64,
}

// would generate a decl-macro like:
macro_rules! __api_call__data {
 ($callback:path, $extra:tt) => {
    $callback!( {$extra},
        // Data serialized with serde_tokenstream::serialize
        { n_fields = 1, field_names = [ field ] }
    }
}}
}

// A user writes a handler like
#[handler(path::to::ApiCall)]
pub fn handle(request: path::to::ApiCall) { ... }

// which, first, expands to:
path::to::__api_call__data!{
   handler_inner,
   {pub fn handle(request: path::to::ApiCall) { ... }}
}

// this expands again to
handler_inner! {
   {pub fn handle(request: path::to::ApiCall) { ... }},
   { n_fields = 1, field_names = [ field ] }
}

// At which point, implementation of handler_inner uses serde_tokenstream::deserialize to get the data back.

Have you thought about implementing this? Alternatively, would you accept such a contribution?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing serde_tokenstream deserialization API and its proc_macro/proc_macro2::TokenStream support. Define the intended reverse operation and round-trip behavior from the issue's examples, then verify that serialized data can be consumed by deserialize in proc-macro contexts. Done means the library provides an accepted serialization contribution with coverage for the described token-tree data.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
compilers, developer-experience
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.