binance / binance/binance-public-data
[UM futures metrics] Silent create_time labeling change on 2026-06-25 (end-of-period -> start-of-period), duplicate 00:00 label across consecutive daily files
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 613
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
The USD-M futures `metrics` daily archives silently changed their `create_time` labeling convention at **2026-06-25 00:00 UTC**, from *end-of-period* labels to *start-of-period* labels. The change shifts the meaning of every row by 5 minutes, applies to all six data columns, and is not mentioned in any changelog or README. The REST API documentation for the corresponding `/futures/data/*` endpoints still describes `timestamp` as *"End time of the period"*, which no longer matches observed behavior.
### What changed
| | before 2026-06-25 | from 2026-06-25 |
|---|---|---|
| daily zip label range | `00:05:00` … `00:00:00` next day | `00:00:00` … `23:55:00` |
| stock columns (`sum_open_interest`, `sum_open_interest_value`, `count_toptrader_long_short_ratio`, `sum_toptrader_long_short_ratio`, `count_long_short_ratio`) at `create_time=T` | snapshot taken **at T** | snapshot taken **at T+5min** |
| `sum_taker_long_short_vol_ratio` at `create_time=T` | flow over **[T−5min, T)** | flow over **[T, T+5min)** |
The new labeling matches the convention the COIN-M metrics archives have used all along (CM daily zips are labeled `00:00 … 23:55` at least back to 2025), so this looks like a UM→CM harmonization — possibly related to the CM-UM integration effective 2026-06-30 — but we found no announcement covering the metrics archives.
### Evidence
**1. Label-set flip at the day boundary (trivially reproducible):**
```python
import io, zipfile, urllib.request, pandas as pd
def labels(day):
u = f"https://data.binance.vision/data/futures/um/daily/metrics/BTCUSDT/BTCUSDT-metrics-{day}.zip"
z = zipfile.ZipFile(io.BytesIO(urllib.request.urlopen(u).read()))
d = pd.read_csv(io.BytesIO(z.read(z.namelist()[0])))
ct = pd.to_datetime(d.create_time)
return ct.min(), ct.max(), len(d)
print(labels("2026-06-24")) # ('2026-06-24 00:05:00', '2026-06-25 00:00:00', 288) <- old
print(labels("2026-06-25")) # ('2026-06-25 00:00:00', '2026-06-25 23:55:00', 288) <- new
```
Because the 06-24 file is end-labeled and the 06-25 file is start-labeled, **both files contain a row labeled `2026-06-25 00:00:00`, with different values** — the old file's row is the snapshot at 00:00, the new file's row is the snapshot at 00:05. We checked 180 symbols: in ≥127/129 symbols with both files present, the two same-labeled rows differ in every column by exactly the magnitude of one normal 5-minute step (they would be identical if the column's convention had not changed).
**2. Taker column content, verified against public aggTrades.** `sum_taker_long_short_vol_ratio` can be reconstructed from the public aggTrades archives (taker buy volume / taker sell volume over a 5-minute window). Scanning window offsets:
- 2024-06, 2025-06, and every month through 2026-06-20: archive row `T` matches the window **[T−5min, T)** with r = 0.998–1.000; the [T, T+5min) window gives r ≈ 0.0–0.2.
- From 2026-06-25 on: archive row `T` matches **[T, T+5min)** with r = 0.998–1.000, and [T−5min, T) drops to noise.
- Day-level pinning: 2026-06-24 is fully old-convention, 2026-06-25 is new-convention from 00:00 (checked in 6-hour sub-windows). Verified on BTCUSDT, ETHUSDT, SOLUSDT, RVNUSDT.
**3. Open-interest content, verified against an independent 5-second poll of `/fapi/v1/openInterest`** (Tardis derivative_ticker, free first-of-month data):
- 2026-06-01: archive `sum_open_interest` at `create_time=T` matches the instantaneous OI **at T** (median relative difference 2.2e-5) vs 3.3e-4 when aligned at T+5min.
- 2026-07-01: reversed — 1.5e-4 at **T+5min** vs 4.1e-4 at T.
### Impact
- Any pipeline that assumed a fixed `create_time` convention now carries a silent 5-minute content shift starting 2026-06-25 — e.g. features sampled "as of row T" now embed data from T+5min that was not observable at T.
- The duplicated `2026-06-25 00:00:00` label (one row in each of two consecutive daily files, different values) breaks naive de-duplication.
- The REST documentation for `/futures/data/*` (`Long/Short Ratio`, `Taker Buy/Sell Volume`, etc.) still says `timestamp` is *"End time of the period, in milliseconds"* — inconsistent with current REST and archive behavior.
### Questions
1. Can you confirm the convention change and its exact effective time (we measure 2026-06-25 00:00 UTC for UM)?
2. Will this be documented in the changelog / README of the metrics dataset, and will the REST endpoint docs be updated to match the current behavior?
3. Which convention is authoritative going forward — is a revert possible? Downstream consumers need this to be stable to interpret `create_time` at all.
4. Is the duplicated `2026-06-25 00:00:00` row intentional (transition artifact), and which of the two rows should be considered canonical?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the 2026-06-24 and 2026-06-25 archive comparison with the Python snippet in the issue, then inspect the repository's README, changelog, and documentation for the futures/data endpoints. Confirm the convention and transition handling with maintainers; done means the authoritative behavior, duplicate-row handling, and matching documentation are agreed and recorded.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- api, data, documentation
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100