google-gemini / google-gemini/genai-processors
Add LowercaseTextProcessor for text normalization
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.1k
- Forks
- 225
- Avg merge
- 1h 56m
- Merged PRs (30d)
- 1
Description
Description of the feature request:
It would be helpful to have a simple processor in genai_processors/contrib that converts all text in incoming ProcessorParts to lowercase. This would make it easier for users to build normalization pipelines.
Proposed API:
Location: genai_processors/contrib/lowercase_text_processor.py
Class: LowercaseTextProcessor
Inherits from: PartProcessor
Logic: If the part is text (is_text(part.mimetype)), convert to lowercase; else, yield unchanged. All metadata is preserved.
What problem are you trying to solve with this feature?
-
Tokenization might use "Hello", "hello", and "HELLO" as different number of tokens. Lowercasing ensures that "Hello", "hello", and "HELLO" are treated the same.
-
Improved search and matching
Any other information you'd like to share?
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing PartProcessor interface and the is_text helper, then inspect related processors under genai_processors/contrib. Implement the proposed class in genai_processors/contrib/lowercase_text_processor.py. Done means text parts are lowercased, non-text parts are unchanged, and all metadata is preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100