microsoft / microsoft/rag-experiment-accelerator
RAG Pattern for Multi-Lingual Scenarios
Open
@auyidi1 is already working on this.
Since Oct 10, 2023.
- Dominant language
- Python
- Stars
- 311
- Forks
- 111
- PR merge metrics
- No merged PRs in 30d
Description
Azure Cognitive Search has skillsets for language detection/processing. Exploration will be required to determine the best implementation. Should experiment with German, Italian, and English as languages to be tested, as these are currently being used in an active customer engagement.
### Tasks
- [x] Determine if the default Standard Lucene language analyzer is sufficient
- [x] Determine support guidance for index creation and querying (e.g. blended or language-specific indexes)
- [x] Add LanguageDetectionSkill support
- [x] Create language analyzer settings `analyzers`, `tokenizers`, `token_filters`, etc to `search_config.json` and update SearchIndexClient settings
- [x] Set data ingestion limits/constraints on chunk size to adhere to maximum record size (i.e. 50k characters) - see `search_config.json`.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.