Improve support for serializing data containing offsets (FilePtr, etc.)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 853
- Forks
- 56
- PR merge metrics
- No merged PRs in 30d
Description
Here's what roblabla had to say (on discord):
actually with seek+write, fileptr seems doable with some global context. Here's the idea: when you get a FilePtr, record the current location, write some zeroes, and schedule a sort of "late serialization" pass. When the structure is fully serialized, all scheduled late serialization pass will run sequentially. That serialization pass will just serialize the underlying structure, seek to the previously saved location, and write the proper offset
that does mean you'd need some sort of heap allocation support to keep track of the serialization passes
but I can't think of a better way.
I had a similar idea awhile back (though I wasn't looking to implement at the library level) but i don't think this'll quite work for some types of files -- specifically those which break it into different sections, and different things go into different sections. i.e.:
struct File {
info: Vec<Info>,
data: DataSection
}
struct DataSection {
// header...
// data blocks go here
}
struct InfoSection {
// header...
list: Vec<Info>,
// string blocks go here
}
struct Info {
data_name: FilePtr32<String>, // points into InfoSection
data_ptr: FilePtr32<Vec<u8>> // points into DataSection
}
under the scheme of the current idea i don't think we have a good way to represent this: we'd write out info section, info, info, info, data section, string, data, string, data, string, data instead of info section, info, info, info, string, string, string, data section, data, data, data. I think this is fixable by adding some sort of marker type Pool with some way of specifying a identifier such that you can do the same thing as the original idea, but for each pool in turn as you come across them while writing the file.
Are there any other methods we might want to consider?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reviewing the existing FilePtr serialization design and the sectioned-layout example, then compare deferred serialization with the proposed Pool approach. Done means selecting and documenting an implementation direction that supports offsets across separate sections.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100