chroma-core / chroma-core/chroma
[Feature Request]: function vectorizer and FileLoader
- Dominant language
- Rust
- Stars
- 29.3k
- Forks
- 2.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 38
Description
### Describe the problem
I find that most of the time, I already have the data I want to vectorized stored somewhere --
therefore copying it over to chromadb is not only wasteful, but also exposes my system to a bunch of sync-maintenance nightmares.
Looking for a solution, I found this thing called [data loaders](https://docs.trychroma.com/multi-modal#data-loaders)
that, if specified, will by applied to the `uris` to get `documents` (contents).ConnectionResetError
The documentation doesn't make it obvious how I should make a data loader, but found
the [DataLoader](https://github.com/chroma-core/chroma/blob/185c76a5ed932aa4b71ea6060474c39014cd4368/chromadb/api/types.py#L203)
in the code, and (a single!) example of one
([ImageLoader](https://github.com/chroma-core/chroma/blob/185c76a5ed932aa4b71ea6060474c39014cd4368/chromadb/utils/data_loaders.py#L9))
in some "data_loaders.py" module.
Apparently, it should be a callable that takes uris and returns their contents
(which I guess will be interpreted as `documents`).
Firstly: I would say that a bit more documentation on this would be useful.
Second: Having a few more ready-to-use (and possibly parametrize) data loaders would be useful.
Last: though I understand that using a data_loader that applies to an iterable of `uris` makes
it easier to create optimized data loaders, I find that in practice, I'll have a function
that takes a single uri and returns a document, and I want to use that function as a data_loader.
### Describe the proposed solution
As far as my last point: I created a `vectorize` function that will create a vectorized version of
a "single item" data loader.
```python
from functools import partial
def vectorize(func, iterable=None):
"""Like builtin map, but returns a list,
and if iterable is None, returns a partial function that can directly be applied
to an iterable.
Example:
>>> vectorize(lambda x: x**2, [1,2,3])
[1, 4, 9]
>>> vectorized_square = vectorize(lambda x: x**2)
>>> vectorized_square([1,2,3])
[1, 4, 9]
"""
if iterable is None:
return partial(vectorize, func)
return list(map(func, iterable))
```
In fact, this `vectorize` function could be useful beyond data loaders only.
We're dealing with a vector database here, and one might find more than one occasion to want to get a vectorized version of some function (for example, to get ids from uris, etc.).
Regarding the second point, I've created a `FileLoader` that will load local (text) files by default, but has some a few parameters to easily make many other types of loaders (relative paths, without extension, from remote url, from pdf file, from s3, from a DB...)
### Alternatives considered
We could also have the inner-mechanism of chromadb accept "single item" data loaders, and dynamically transform them to their vectorized counterpart. This could be seen as a departure from the "explicit over implicit" principle though, so should only be done if there's no ambiguity possible.
### Importance
would make my life easier
### Additional Information
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.