michaelfeil / michaelfeil/infinity

Reranker dynamic quantization

Open
#363 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Python
Stars
2.9k
Forks
206
PR merge metrics
No merged PRs in 30d

Description

### Feature request

--dtype support for rerankers

### Motivation

Easily quantize cross encoder models

### Your contribution

. (will look into this)

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name any files, tests, or entry points. Start by locating the reranker and cross-encoder model handling, then trace how model dtype is selected and whether dynamic quantization is already supported elsewhere. Done means rerankers accept the requested dtype and quantized cross-encoder behavior is covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.