binance / binance/binance-public-data
[Futures data quality] Missing and duplicate USD-M metrics/markPriceKlines timestamps for BTCUSDT and ETHUSDT (2023-2025)
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 613
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
I found reproducible continuity defects in official Binance USD-M public-data archives for BTCUSDT and ETHUSDT. The affected local files match the currently published official object metadata and `.CHECKSUM` SHA-256 values, so this does not appear to be a local corruption or incomplete download.
Audit date: 2026-07-31 UTC
Submission date: 2026-08-03 UTC
Source: `data.binance.vision`
Period checked: 2023-01-01 through 2025-12-31
Symbols: BTCUSDT and ETHUSDT
### 1. Mark-price kline gap affecting both symbols
Affected objects:
```text
data/futures/um/daily/markPriceKlines/BTCUSDT/5m/BTCUSDT-5m-2023-11-10.zip
data/futures/um/daily/markPriceKlines/ETHUSDT/5m/ETHUSDT-5m-2023-11-10.zip
```
Both archives contain 283 data rows instead of the expected 288 five-minute rows.
The timestamps jump from:
```text
2023-11-10T03:35:00Z
```
directly to:
```text
2023-11-10T04:05:00Z
```
Missing from both symbols:
```text
2023-11-10T03:40:00Z
2023-11-10T03:45:00Z
2023-11-10T03:50:00Z
2023-11-10T03:55:00Z
2023-11-10T04:00:00Z
```
BTCUSDT currently published evidence:
```text
content_length=7609
last_modified=2023-11-13T01:41:46Z
etag="38b1cbefcfc8af805534973da30a20c2"
official_sha256=c16d3d0f7f3110e16d7974f3f00f8cdeb0d13b9252ef70225aff3db88b846a79
local_sha256=c16d3d0f7f3110e16d7974f3f00f8cdeb0d13b9252ef70225aff3db88b846a79
```
ETHUSDT currently published evidence:
```text
content_length=7304
last_modified=2023-11-13T01:41:54Z
etag="c9471d172680f4f8b4198ce6f0f69637"
official_sha256=4099127b68ff2874f424d05c4bdc11f44598b4f021b9b5baa3aa24b92c0ea743
local_sha256=4099127b68ff2874f424d05c4bdc11f44598b4f021b9b5baa3aa24b92c0ea743
```
For comparison, the ordinary USD-M 5-minute contract kline archives and metrics archives for the same date contain the expected timestamps in that interval.
### 2. Metrics continuity and duplicate-timestamp defects
Dataset families checked:
```text
data/futures/um/daily/metrics/BTCUSDT/
data/futures/um/daily/metrics/ETHUSDT/
```
Expected accepted grid per symbol:
```text
2023-01-01T00:05:00Z through 2025-12-31T23:55:00Z
frequency=5 minutes
expected_rows=315647
```
Observed summary before 2026-01-01 UTC:
| Symbol | Rows before 2026 | Unique timestamps before 2026 | Missing expected timestamps | Duplicate timestamps |
|---|---:|---:|---:|---:|
| BTCUSDT | 315514 | 315512 | 333 | 2 |
| ETHUSDT | 315514 | 315512 | 300 | 2 |
Largest common missing intervals include:
```text
2024-02-16T13:35:00Z through 2024-02-16T23:55:00Z — 125 five-minute timestamps
2025-07-22T13:45:00Z through 2025-07-22T16:30:00Z — 34 five-minute timestamps
2025-07-21T16:25:00Z through 2025-07-21T17:10:00Z — 10 five-minute timestamps
2023-09-12T08:40:00Z through 2023-09-12T08:50:00Z — 3 five-minute timestamps
```
Duplicate timestamps observed for both symbols:
```text
2024-04-08T00:00:00Z
2024-05-01T00:00:00Z
```
The archives also include `2023-01-01T00:00:00Z`, although the regular five-minute sequence used by the dataset begins at `00:05:00Z`.
### 3. Integrity checks performed
- Generated the full expected official object set for the period.
- Acquired and parsed 6,648 matching `.CHECKSUM` files.
- Downloaded 6,648 authorized ZIP archives.
- Recomputed SHA-256 for all 6,648 ZIP archives.
- All local hashes matched the official checksum files.
- No ZIP download, size, checksum, or safety failures occurred.
- The frozen manifest SHA-256 is:
```text
0bb195963bae545b303074b2953c38977b036522ff662effa57308176aac4dba
```
### 4. Minimal reproduction
For the mark-price defect:
1. Download either affected archive and its `.CHECKSUM` file.
2. Verify the ZIP against the official SHA-256.
3. Read the single CSV member.
4. Parse `open_time` as UTC epoch milliseconds.
5. Compare consecutive timestamps against a five-minute grid.
6. Observe the jump from `03:35` to `04:05` UTC.
For metrics:
1. Download all daily metrics ZIPs for the fixed symbol and period.
2. Verify each ZIP with the matching official `.CHECKSUM`.
3. Parse `create_time` as UTC.
4. Compare unique timestamps against the complete five-minute grid.
5. Record missing and duplicate timestamps.
### Request
Could Binance please:
1. confirm whether these gaps and duplicates are known;
2. replace the affected archive objects and `.CHECKSUM` files if corrected source data are available;
3. publish an official update/changelog identifying all replaced objects and old/new checksums;
4. clarify whether gaps in the `metrics` dataset can be expected by design or should be treated as archive defects?
I can provide the complete object manifest, download ledger, and deterministic scanner output if useful.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the affected markPriceKlines archives for BTCUSDT and ETHUSDT and the metrics archives under data/futures/um/daily/metrics/. Reproduce the UTC five-minute continuity scan and verify the matching .CHECKSUM files against the reported hashes. Done means confirming whether the missing and duplicate timestamps are archive defects or expected behavior, and documenting any corrected objects and checksum changes.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100