anomalyco / anomalyco/models.dev

Feature Request: Add logprobs sampling metadata to model database

Open
#2,743 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
6.9k
Forks
1.7k
Avg merge
3h 21m
Merged PRs (30d)
873

Description

Feature Request: Add logprobs metadata to model database

Why we need logprobs data

Logprobs are a natural model fingerprint — different models produce distinct token probability distributions on the same prompt, and these distributions are inherently difficult to forge. If an API claims to serve Model A but returns logprobs inconsistent with Model A's baseline, users can immediately detect misrepresentation.

Current problems
  1. No central reference: There's no public dataset of per-model logprobs baselines. Users who suspect mislabeling have nothing to compare against.
  2. High collection cost: Building baselines requires many API calls with consistent prompts — impractical for individual users to do across dozens of models.
  3. models.dev gap: The database already tracks pricing, context limits, and capability flags, but has zero information about logprobs support.
What to add

A logprobs section per model recording:

  • Whether logprobs are available
  • Access method (built-in by default, or requires parameter)

Happy to help collect baselines for a first batch of mainstream models if accepted.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or tests are named in the issue. Start by locating the model database schema and existing capability metadata, then determine how logprobs availability and access method should be represented consistently; done means the accepted schema is populated for the first batch of models and validated by the project's checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.