TimelyDataflow / TimelyDataflow/differential-dataflow
Expose pointer to memory in BatchFrontier/BatchReader/BatchContainer/Cursor?
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3k
- Forks
- 211
- Avg merge
- 10h 42m
- Merged PRs (30d)
- 34
Description
I was exploring the idea of using madvise MADV_SEQUENTIAL to pre-fault batches (for use in Materialize).
However, BatchFrontier/BatchReader only exposes a Cursor.
BatchReader says
A batch of updates whose contents may be read.
This is a restricted interface to batches of updates, which support the reading of the batch’s contents, but do not expose ways to construct the batches. This trait is appropriate for views of the batch, and is especially useful for views derived from other sources in ways that prevent the construction of batches from the type of data in the view (for example, filtered views, or views with extended time coordinates).
Given that, it seems to me that even if there was a function that returned a pointer to the batch storage, that would not necessarily violate the intention of "not exposing a way to construct the batch".
Anyways:
- I was curious (just for my own understanding), what sort of abstraction Batch is supposed to be exactly. (It's quite possible that this is not even the right thing to do from the perspective of making improvements to Materialize...)
- Arguably the ability to madvise things appropriately may be a useful thing purely from the perspective of differential dataflow. I wonder what the right API would be for it, maybe it should be built-in for certain operators that are known to have sequential access patterns instead of just exposing pointers for the users to try and do it themselves, not sure
One option would be something like:
pub trait BatchContainer: 'static {
// ... existing methods ...
/// Returns memory regions backing this container for prefetching.
/// Each (ptr, len) pair represents a contiguous memory region.
fn memory_regions(&self) -> impl Iterator<Item = (*const u8, usize)>;
}
Which could then be appropriately madvised... somewhere. Though looking at OrdValBatch, it is actually not entirely clear to me how reads of keys vs vals vs upds should actually be madvised. (I guess it may depend on the workload...?)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the BatchFrontier, BatchReader, BatchContainer, Cursor, and OrdValBatch APIs referenced in the issue, then inspect how batch memory is represented and accessed. Define whether exposing memory regions or providing built-in madvise support is appropriate, including which regions should be advised; the issue currently has no settled acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100