处理日语长视频时报错
Nobody has claimed this yet.
Assessment
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Newbie friendliness
- 35/100
Research direction
Start in st.py and trace the text-processing path through core/step3_1_spacy_split.py and core/spacy_utils/split_by_mark.py. Reproduce the failure with a long Japanese input and inspect how it reaches the tokenizer. Done means long Japanese videos no longer raise the reported input-length tokenization exception and processing continues.
Written by the indexing model from the issue text.
Description
2024-11-07 21:11:45.306 Uncaught app exception
Traceback (most recent call last):
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\streamlit\runtime\scriptrunner\exec_code.py", line 88, in exec_func_with_error_handling
result = func()
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\streamlit\runtime\scriptrunner\script_runner.py", line 590, in code_to_exec
exec(code, module.dict)
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\st.py", line 117, in
main()
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\st.py", line 113, in main
text_processing_section()
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\st.py", line 30, in text_processing_section
process_text()
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\st.py", line 47, in process_text
step3_1_spacy_split.split_by_spacy()
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\core\step3_1_spacy_split.py", line 17, in split_by_spacy
split_by_mark(nlp)
File "C:\Users\deepf\Desktop\VideoLingo\VideoLingo\core\spacy_utils\split_by_mark.py", line 21, in split_by_mark
doc = nlp(input_text)
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\spacy\language.py", line 1037, in call
doc = self._ensure_doc(text)
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\spacy\language.py", line 1128, in ensure_doc
return self.make_doc(doc_like)
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\spacy\language.py", line 1120, in make_doc
return self.tokenizer(text)
File "C:\Users\deepf\anaconda3\envs\videolingo\lib\site-packages\spacy\lang\ja_init.py", line 56, in call
sudachipy_tokens = self.tokenizer.tokenize(text)
Exception: Tokenization error: Input is too long, it can't be more than 49149 bytes, was 116123
- Dominant language
- Python
- Stars
- 18.5k
- Forks
- 2k
- Avg merge
- 7h 41m
- Merged PRs (30d)
- 12
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Huanshere/VideoLingo
-
Difficulty 1/5 1-3 hours Newbie friendliness 78/100
Huanshere/VideoLingo#587 · 1 reaction ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
Huanshere/VideoLingo#561 · 2 reactions ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
Huanshere/VideoLingo#545 ·
-
Difficulty 5/5 Over a week Newbie friendliness 30/100
Huanshere/VideoLingo#598 · 3 comments ·
-
现在能一键式流程，能不能增加分步骤处理 Open
Difficulty 4/5 3-5 days Newbie friendliness 52/100
Huanshere/VideoLingo#593 · 1 comment ·
All issues in Huanshere/VideoLingo
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100