chroma-core / chroma-core/chroma

[Feature Request]: video Embeddings : chromaDB also for videos

Open
#3,533 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
29.3k
Forks
2.5k
Avg merge
1d 4h
Merged PRs (30d)
38

Description

### Describe the problem

For up until now chromaDB is only for text, from this feature it will be the solution for videos as well

like "Chat With Video" / "Talk To Video"

### Describe the proposed solution

**function that creates embeddings for videos**

### steps :

video ==> audio ==> text ==> nltk(sent_tokenize) ==> vector (steps followed in text ) -- (creating collection for sentences in video )

### sample code :

from moviepy import VideoFileClip
import speech_recognition as sr

def extract_audio_from_video(video_file_path , audio_file_path):
'''

desc : extracts audio from video_file_path and stores it in audio_file_path

input :
video_file_path : path to video file
audio_file_path : path to audio file

output :

'''

video = VideoFileClip(video_file_path)
video.audio.write_audiofile(audio_file_path)

def transcribe_audio_to_text(audio_file_path , text_file_path):
'''
desc : transcribes the audio to text

input :
audio_file_path : path to audio file
text_file_path : path to text file to save the text

output :
'''
recognizer = sr.Recognizer()
with sr.AudioFile(audio_file_path) as source:
audio_data = recognizer.record(source)
text = recognizer.recognize_google(audio_data)
with open(text_file_path , "w") as file:
file.writelines(text)
return text

video_file_path = "samples.mp4"
audio_file_path = "temp_audio.wav"
text_file_path = "text_of_video.txt"

extract_audio_from_video(video_file_path, audio_file_path)
transcription = transcribe_audio_to_text(audio_file_path , text_file_path)
print("Transcription:")
print(transcription)

and I would like to contribute to this feature

### Alternatives considered

_No response_

### Importance

nice to have

### Additional Information

I would like to work on it!

Contributor guide

No contributing guide indexed for this repository

Research direction

No repository files, tests, or entry points are identified. Start by reviewing the existing text embedding and Chroma collection flow, then determine how the proposed video-to-audio-to-text pipeline would fit it. The issue needs an agreed design, dependencies, and acceptance criteria before completion can be measured.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
audio-video-rtc, databases, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.