binance / binance/binance-public-data

[UM futures metrics] Silent create_time labeling change on 2026-06-25 (end-of-period -> start-of-period), duplicate 00:00 label across consecutive daily files

Open
#491 0 comments 0 reactions 0 assignees View on GitHub
Futures
Dominant language
Python
Stars
2.5k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

### Summary

The USD-M futures `metrics` daily archives silently changed their `create_time` labeling convention at **2026-06-25 00:00 UTC**, from *end-of-period* labels to *start-of-period* labels. The change shifts the meaning of every row by 5 minutes, applies to all six data columns, and is not mentioned in any changelog or README. The REST API documentation for the corresponding `/futures/data/*` endpoints still describes `timestamp` as *"End time of the period"*, which no longer matches observed behavior.

### What changed

| | before 2026-06-25 | from 2026-06-25 |
|---|---|---|
| daily zip label range | `00:05:00` … `00:00:00` next day | `00:00:00` … `23:55:00` |
| stock columns (`sum_open_interest`, `sum_open_interest_value`, `count_toptrader_long_short_ratio`, `sum_toptrader_long_short_ratio`, `count_long_short_ratio`) at `create_time=T` | snapshot taken **at T** | snapshot taken **at T+5min** |
| `sum_taker_long_short_vol_ratio` at `create_time=T` | flow over **[T−5min, T)** | flow over **[T, T+5min)** |

The new labeling matches the convention the COIN-M metrics archives have used all along (CM daily zips are labeled `00:00 … 23:55` at least back to 2025), so this looks like a UM→CM harmonization — possibly related to the CM-UM integration effective 2026-06-30 — but we found no announcement covering the metrics archives.

### Evidence

**1. Label-set flip at the day boundary (trivially reproducible):**

```python
import io, zipfile, urllib.request, pandas as pd
def labels(day):
u = f"https://data.binance.vision/data/futures/um/daily/metrics/BTCUSDT/BTCUSDT-metrics-{day}.zip"
z = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(u).read()))
d = pd.read_csv(io.BytesIO(z.read(z.namelist()[0])))
ct = pd.to_datetime(d.create_time)
return ct.min(), ct.max(), len(d)

print(labels("2026-06-24")) # ('2026-06-24 00:05:00', '2026-06-25 00:00:00', 288) <- old
print(labels("2026-06-25")) # ('2026-06-25 00:00:00', '2026-06-25 23:55:00', 288) <- new
```

Because the 06-24 file is end-labeled and the 06-25 file is start-labeled, **both files contain a row labeled `2026-06-25 00:00:00`, with different values** — the old file's row is the snapshot at 00:00, the new file's row is the snapshot at 00:05. We checked 180 symbols: in ≥127/129 symbols with both files present, the two same-labeled rows differ in every column by exactly the magnitude of one normal 5-minute step (they would be identical if the column's convention had not changed).

**2. Taker column content, verified against public aggTrades.** `sum_taker_long_short_vol_ratio` can be reconstructed from the public aggTrades archives (taker buy volume / taker sell volume over a 5-minute window). Scanning window offsets:

- 2024-06, 2025-06, and every month through 2026-06-20: archive row `T` matches the window **[T−5min, T)** with r = 0.998–1.000; the [T, T+5min) window gives r ≈ 0.0–0.2.
- From 2026-06-25 on: archive row `T` matches **[T, T+5min)** with r = 0.998–1.000, and [T−5min, T) drops to noise.
- Day-level pinning: 2026-06-24 is fully old-convention, 2026-06-25 is new-convention from 00:00 (checked in 6-hour sub-windows). Verified on BTCUSDT, ETHUSDT, SOLUSDT, RVNUSDT.

**3. Open-interest content, verified against an independent 5-second poll of `/fapi/v1/openInterest`** (Tardis derivative_ticker, free first-of-month data):

- 2026-06-01: archive `sum_open_interest` at `create_time=T` matches the instantaneous OI **at T** (median relative difference 2.2e-5) vs 3.3e-4 when aligned at T+5min.
- 2026-07-01: reversed — 1.5e-4 at **T+5min** vs 4.1e-4 at T.

### Impact

- Any pipeline that assumed a fixed `create_time` convention now carries a silent 5-minute content shift starting 2026-06-25 — e.g. features sampled "as of row T" now embed data from T+5min that was not observable at T.
- The duplicated `2026-06-25 00:00:00` label (one row in each of two consecutive daily files, different values) breaks naive de-duplication.
- The REST documentation for `/futures/data/*` (`Long/Short Ratio`, `Taker Buy/Sell Volume`, etc.) still says `timestamp` is *"End time of the period, in milliseconds"* — inconsistent with current REST and archive behavior.

### Questions

1. Can you confirm the convention change and its exact effective time (we measure 2026-06-25 00:00 UTC for UM)?
2. Will this be documented in the changelog / README of the metrics dataset, and will the REST endpoint docs be updated to match the current behavior?
3. Which convention is authoritative going forward — is a revert possible? Downstream consumers need this to be stable to interpret `create_time` at all.
4. Is the duplicated `2026-06-25 00:00:00` row intentional (transition artifact), and which of the two rows should be considered canonical?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the 2026-06-24 and 2026-06-25 archive comparison with the Python snippet in the issue, then inspect the repository's README, changelog, and documentation for the futures/data endpoints. Confirm the convention and transition handling with maintainers; done means the authoritative behavior, duplicate-row handling, and matching documentation are agreed and recorded.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
api, data, documentation
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.