ClickHouse / ClickHouse/clickhouse-rs
Support custom serialization types in 0.14
- Dominant language
- Rust
- Stars
- 559
- Forks
- 172
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 3
Description
### Use case
We have a complex data structure we serialize to Clickhouse. The in memory format doesn't exactly match the row format so we use a wrapper with a customer serde Serialize implementation to avoid memory overhead of having to keep two copies in different formats in memory.
In 0.13 this worked great because we could manually implement Row and implement Serialize to emit the columns dynamically. In 0.14 we can no longer implement Row because RowKind is private. The Row macro requires your structure fields to match the output, so we can't use this with our custom serializer.
Without this support we're stuck on 0.13 due to the memory overhead of having to copy into a new structure not working for our high throughput data pipeline.
### Describe the solution you'd like
Ideally expose RowKind so we can implement Row again. I understand this is subject to breaking changes based on documentation and it's hidden, but we're more than willing to fix up breaks each version and don't require a contract that avoids changes. This would match the 0.13 behavior.
### Describe the alternatives you've considered
If the Row derive macro had a mode where we could specify column names and that we're doing custom serialization that would also work great and possibly avoid exposing the internal types.
### Additional context
Here's a simplified example of what we're doing today. The real structure is more like a graph hence the need for a different in memory layout vs row format:
```rust
/// Wrapper for serializing
pub struct SerializationWrapper<'a>
{
/// In memory object
pub data: &'a MyStruct,
}
impl Row for SerializationWrapper <'_>
{
const COLUMN_NAMES: &'static [&'static str] = &[
"col1",
"col2",
];
}
impl Serialize for SerializationWrapper <'_>
{
fn serialize(&self, serializer: S) -> Result
where
S: serde::Serializer,
{
let mut s = serializer.serialize_struct("mystruct", 2)?;
s.serialize_field("col1", &data.items.len())?;
s.serialize_field("col2", &data.field2)?;
s.end()
}
}
```
Contributor guide
Research direction
Start by locating the 0.14 Row trait, private RowKind, and Row derive macro entry points, then compare them with the described 0.13 behavior. Review how custom Serialize implementations are accepted and determine whether exposing RowKind or adding a custom-serialization macro mode is the appropriate direction. Done means a wrapper with a different in-memory layout can serialize its declared ClickHouse columns without copying data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100