Feature Request: A/B tests for prompts
Open
Nobody has claimed this yet.
enhancement
team/llm-analytics
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 7h 27m
- Merged PRs (30d)
- 222
Description
Feature request
Is your feature request related to a problem?
PostHog now has prompt management - users would like to run A/B tests to experiment on changes to their prompts.
Additional context
From: https://posthoghelp.zendesk.com/agent/tickets/55904 (moved to PostHog: https://us.posthog.com/project/2/support/tickets/57504)
Debug info
- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start with the linked prompt management documentation and the referenced support tickets, then trace how prompt changes and experiments are currently represented; the feature is done when a concrete scope for running and evaluating prompt A/B tests is agreed and implemented.
Written by the indexing model from the issue text.
Assessment
- Domain
- ai, analytics
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100