open-feature / open-feature/spec

Official evaluation metrics hook for experimentation

Open
#371 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
1.3k
Forks
58
Avg merge
2d 18h
Merged PRs (30d)
2

Description

Context

At KubeCon EU 2026, the experimentation discussion (recap) identified interest in having the SDK produce basic evaluation metrics that analytics vendors can consume for experimentation analysis.

Problem

Today, if you want evaluation metrics (e.g. how many evaluations per flag key, broken down by targeting reason, how many unique targeting keys have seen a given variant), you have to build that yourself. Multiple participants noted that this is a common need across experimentation platforms and the data is straightforward to collect.

Proposal

Define an official evaluation metrics hook that collects basic evaluation data:

  • Evaluation count per flag key
  • Evaluations broken down by reason (e.g. TARGETING_MATCH, DEFAULT, STATIC)
  • Unique targeting keys per flag/variant
  • Variant distribution per flag key

This hook could be published as an official OpenFeature contrib package. Analytics vendors (or an OFREP endpoint) could then consume this data without each needing to implement their own collection logic.

The hook approach is preferred over new API surface since this is buildable with the existing hook model.

Questions

  • Should this be a spec recommendation or just an official contrib hook?
  • What is the minimal set of metrics that provides value across vendors?
  • Should this hook emit OTel metrics, or define its own output interface that vendors implement?

Related

  • #370: Experimentation support: standardized context fields and experiment grouping
  • #276: Implement Tracking in SDKs

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing hook model, then review related issues #370 and #276 for context on experimentation and tracking. Done requires agreement on the minimal evaluation metrics, whether this belongs in the specification or an official contrib package, and whether the output should use OpenTelemetry or a separate interface.

Written by the indexing model from the issue text.

Assessment

Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.