chroma-core / chroma-core/chroma

[Feature Request]: function vectorizer and FileLoader

Open
#1,606 2 comments 0 reactions 1 assignee Claimed by @atroyn View on GitHub
enhancement
Dominant language
Rust
Stars
29.3k
Forks
2.5k
Avg merge
1d 4h
Merged PRs (30d)
38

Description

### Describe the problem

I find that most of the time, I already have the data I want to vectorized stored somewhere --
therefore copying it over to chromadb is not only wasteful, but also exposes my system to a bunch of sync-maintenance nightmares.

Looking for a solution, I found this thing called [data loaders](https://docs.trychroma.com/multi-modal#data-loaders)
that, if specified, will by applied to the `uris` to get `documents` (contents).ConnectionResetError
The documentation doesn't make it obvious how I should make a data loader, but found
the [DataLoader](https://github.com/chroma-core/chroma/blob/185c76a5ed932aa4b71ea6060474c39014cd4368/chromadb/api/types.py#L203)
in the code, and (a single!) example of one
([ImageLoader](https://github.com/chroma-core/chroma/blob/185c76a5ed932aa4b71ea6060474c39014cd4368/chromadb/utils/data_loaders.py#L9))
in some "data_loaders.py" module.

Apparently, it should be a callable that takes uris and returns their contents
(which I guess will be interpreted as `documents`).

Firstly: I would say that a bit more documentation on this would be useful.

Second: Having a few more ready-to-use (and possibly parametrize) data loaders would be useful.

Last: though I understand that using a data_loader that applies to an iterable of `uris` makes
it easier to create optimized data loaders, I find that in practice, I'll have a function
that takes a single uri and returns a document, and I want to use that function as a data_loader.

### Describe the proposed solution

As far as my last point: I created a `vectorize` function that will create a vectorized version of
a "single item" data loader.

```python
from functools import partial

def vectorize(func, iterable=None):
"""Like builtin map, but returns a list,
and if iterable is None, returns a partial function that can directly be applied
to an iterable.

Example:

>>> vectorize(lambda x: x**2, [1,2,3])
[1, 4, 9]
>>> vectorized_square = vectorize(lambda x: x**2)
>>> vectorized_square([1,2,3])
[1, 4, 9]
"""
if iterable is None:
return partial(vectorize, func)
return list(map(func, iterable))
```

In fact, this `vectorize` function could be useful beyond data loaders only.
We're dealing with a vector database here, and one might find more than one occasion to want to get a vectorized version of some function (for example, to get ids from uris, etc.).

Regarding the second point, I've created a `FileLoader` that will load local (text) files by default, but has some a few parameters to easily make many other types of loaders (relative paths, without extension, from remote url, from pdf file, from s3, from a DB...)

### Alternatives considered

We could also have the inner-mechanism of chromadb accept "single item" data loaders, and dynamically transform them to their vectorized counterpart. This could be seen as a departure from the "explicit over implicit" principle though, so should only be done if there's no ambiguity possible.

### Importance

would make my life easier

### Additional Information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.