anomalyco / anomalyco/models.dev
Feature Request: Add logprobs sampling metadata to model database
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 6.9k
- Forks
- 1.7k
- Avg merge
- 3h 21m
- Merged PRs (30d)
- 873
Description
Feature Request: Add logprobs metadata to model database
Why we need logprobs data
Logprobs are a natural model fingerprint — different models produce distinct token probability distributions on the same prompt, and these distributions are inherently difficult to forge. If an API claims to serve Model A but returns logprobs inconsistent with Model A's baseline, users can immediately detect misrepresentation.
Current problems
- No central reference: There's no public dataset of per-model logprobs baselines. Users who suspect mislabeling have nothing to compare against.
- High collection cost: Building baselines requires many API calls with consistent prompts — impractical for individual users to do across dozens of models.
- models.dev gap: The database already tracks pricing, context limits, and capability flags, but has zero information about logprobs support.
What to add
A logprobs section per model recording:
- Whether logprobs are available
- Access method (built-in by default, or requires parameter)
Happy to help collect baselines for a first batch of mainstream models if accepted.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files or tests are named in the issue. Start by locating the model database schema and existing capability metadata, then determine how logprobs availability and access method should be represented consistently; done means the accepted schema is populated for the first batch of models and validated by the project's checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100