Metaculus / Metaculus/metaculus

MC recency weighted bug when deleting forecasts

Open
#4,157 1 comment 0 reactions 1 assignee View on GitHub

@SylvainChevalier is already working on this.

Since Jan 24, 2026.

Medium Priority Stale
Dominant language
TypeScript
Stars
113
Forks
34
Avg merge
16h 34m
Merged PRs (30d)
30

Description

For an MC question when deleting or adding an option - this triggers an automatic forecast for all users at the exact moment of the deletion or the exact moment of the end of the grace period.
This means that the recency-weighting algorithm no longer respects which user forecast was actually made first.
e.g.
user 1 forecasts at t0
user 2 forecasts at t1
option deleted at t2
user 1 and user 2 get automatic forecasts at t2 made on their behaves representing their previous forecast but with the deleted option's probability integrated into the catchall
upon calculation of the CP at t2, user 1's forecast should be considered older, so it should be weighted less than user 2's, but since they both get automatic forecasts at t2, their "order" will be arbitrary so recency weighting CP won't represent the value that it should

Unsure about the best solution here. It's a relatively minor bug as the worst failure mode is a high profile question with lots of outdated forecasts that get "refreshed" incorrectly. Hopefully we can solve this before that happens (e.g. us election primary candidate prediction).

The niave solution would be to detect forecast.source == "automatic" when calculating aggregations and take the age of the forecast to be the start of the previous forecast. Extra tech debt inside the aggregation function, but might be the only reliable way to squash this issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.