huggingface / huggingface/candle

Improve extracting values from `gguf_file::Value`

Open
#2,245 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

I have been working on improving `Value` conversion within `mistral.rs`, but if `candle` were to improve direct support here it'd probably make it less necessary, and may benefit others?

Perhaps something like `rkyv` for serialize/deserialize could be used, I haven't looked into that.

---

In `mistral.rs` since `Value` is a foreign type, I cannot impl `From` / `TryFrom`, so I've had to workaround that:

```rs
trait TryFromValue {
fn try_from_value(value: gguf_file::Value) -> Result where Self: Sized;
}

// Value wrapped types, each has a different conversion method:
// akin macro used to minimize repetition for supporting each variant:
akin! {
let &types = [String, f32, u32];
let &to_type = [value.to_string().cloned(), value.to_f32(), value.to_u32()];

impl TryFromValue for *types {
fn try_from_value(value: gguf_file::Value) -> Result {
*to_type.or_else(|_| candle_core::bail!("value is not a `*types`"))
}
}
}

// Vec to Vec from above types:
impl TryFromValue for Vec {
fn try_from_value(value_vec: gguf_file::Value) -> Result {
value_vec.to_vec().or_else(|_| candle_core::bail!("value is not a `Vec`"))?.clone()
.into_iter()
.map(|item| T::try_from_value(item))
.collect()
}
}

trait TryValueInto: Sized {
fn try_value_into(self) -> Result;
}

impl TryValueInto for gguf_file::Value {
fn try_value_into(self) -> Result {
T::try_from_value(self)
}
}

impl TryValueInto for Option {
fn try_value_into(self) -> Result {
match self {
Some(value) => value.try_value_into(),
None => candle_core::bail!("Option is missing value"),
}
}
}

struct MetadataContext<'a> {
path_prefix: String,
metadata: &'a HashMap
}

impl MetadataContext<'_> {
// Retrieve a prop the struct needs by querying the metadata content:
fn get_value(self, field_name: &str) -> Result {
let prop_key = format!("{prefix}.{field_name}", prefix = self.path_prefix);
let value = self.metadata.get(&prop_key).cloned();

// Unwrap the inner value of the `Value` enum via trait method,
// otherwise format error with prop key as context:
value.try_value_into().or_else(|e| candle_core::bail!("`{prop_key}` `{e}`"))
}
}
```

I can then more easily grab the values needed:

```rs
struct PropsGGUF {
model: String,
tokens: Vec,
added_tokens: Option>,
scores: Option>,
merges: Option>,
unk: Option,
eos: u32,
bos: u32,
}

impl TryFrom> for PropsGGUF {
type Error = anyhow::Error;

// A deserializer derive would be more convenient?:
fn try_from(c: MetadataContext) -> Result {
let tokenizer_ggml = PropsGGUF {
model: c.get_value("model")?,
tokens: c.get_value("tokens")?,
added_tokens: c.get_value("added_tokens").ok(),
scores: c.get_value("scores").ok(),
merges: c.get_value("merges").ok(),
unk: c.get_value("unknown_token_id").ok(),
eos: c.get_value("eos_token_id")?,
bos: c.get_value("bos_token_id")?,
};

Ok(tokenizer_ggml)
}
}
```

And use it:

```rs
let metadata = MetadataContext {
path_prefix: "tokenizer.ggml".to_string(),
metadata: &content.metadata
};
let props = PropsGGUF::try_from(metadata)?;

let PropsGGUF {
model,
tokens,
added_tokens,
..
} = props;

// ...

let PropsGGUF {
unk,
eos,
bos,
..
} = props;
```

Perhaps those traits would be helpful upstream? (_`get_value()` is more of a helper for `mistral.rs`_)

I put it together to minimize the need to parse `Value` like this in `mistral.rs`:

```rs
let added_tokens = content
.metadata
.get("tokenizer.ggml.added_tokens")
.map(|items| {
items
.to_vec()
.expect("GGUF tokenizer added_tokens is not a vec.")
.iter()
.map(|t| {
t.to_string()
.expect("GGUF added_token is not a string.")
.clone()
})
.collect::>()
});
```

---

**FWIW:** I also heard that `to_string()` and `to_vec()` are incorrectly named, they should rather be `as_string()` / `as_vec()` since these two return references that would require `clone()` to get the implied owned type?:

https://github.com/huggingface/candle/blob/cd4d941ed10fd334333cf5793e311d2bef88a438/candle-core/src/quantized/gguf_file.rs#L255-L267

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.