binance / binance/binance-public-data

Missing USD-M 1m markPriceKlines and indexPriceKlines rows across multiple symbols

Open
#483 0 comments 0 reactions 0 assignees View on GitHub
Futures
Dominant language
Python
Stars
2.5k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

Hello Binance public-data maintainers,

While validating the official USD-M Futures 1-minute mark and index price
archives, I found synchronized missing candle rows across multiple symbols.

## Affected data

- Market: USD-M Futures (`data/futures/um`)
- Data types: `markPriceKlines`, `indexPriceKlines`
- Interval: `1m`
- Symbols: `BTCUSDT`, `ETHUSDT`, `XRPUSDT`, `LINKUSDT`, `SOLUSDT`, `DOGEUSDT`
- Missing rows: **603 symbol/data-type minutes**
- `markPriceKlines`: 305
- `indexPriceKlines`: 298
- Distinct missing open times: **66**
- Contiguous symbol/data-type gap ranges: **48**

| UTC date | Missing symbol/data-type minutes | Distinct missing open times |
|---|---:|---:|
| 2020-01-19 | 232 | 29 |
| 2020-12-17 | 288 | 24 |
| 2022-07-12 | 58 | 10 |
| 2022-07-13 | 1 | 1 |
| 2024-08-12 | 24 | 2 |

## Checks performed

1. Checked 43 affected official monthly ZIP files and verified
their published `.CHECKSUM` files.
2. Checked the corresponding official daily ZIP files:
40 were downloaded with verified checksums and
4 returned `NOT_AVAILABLE`.
3. Queried the exact 48 missing ranges through the public
`/fapi/v1/markPriceKlines` or `/fapi/v1/indexPriceKlines` endpoint. Every
response was the empty JSON array `[]`.
4. Re-ran Binance's own `download-futures-markPriceKlines.py` and
`download-futures-indexPriceKlines.py` scripts at repository commit
`5c7f3197591c0d54d85dc43066226bc4c671d47a` for all affected dates. The scripts downloaded and verified
52 ZIP files, but
recovered **0 of the 603 missing rows**.
5. No interpolation or third-party values were used.

The synchronized gaps include:

- 2020-01-19 13:09–13:37 UTC
- 2020-12-17 07:32–07:55 UTC
- scattered minutes on 2022-07-12 and 2022-07-13
- 2024-08-12 10:02–10:03 UTC

## Attachments

[binance_usdm_mark_index_missing_minutes.csv](https://github.com/user-attachments/files/30496219/binance_usdm_mark_index_missing_minutes.csv)
[binance_usdm_mark_index_gap_ranges.csv](https://github.com/user-attachments/files/30496231/binance_usdm_mark_index_gap_ranges.csv)
[evidence_manifest.json](https://github.com/user-attachments/files/30496234/evidence_manifest.json)
[official_downloader_probe.json](https://github.com/user-attachments/files/30496237/official_downloader_probe.json)

- `binance_usdm_mark_index_missing_minutes.csv`: every missing symbol,
data type, and 1-minute open time
- `binance_usdm_mark_index_gap_ranges.csv`: surrounding present timestamps,
archive URLs/statuses/checksums, exact public API URLs, and empty-response
SHA256 evidence
- `evidence_manifest.json`: counts and SHA256 hashes for the two CSV files
- `official_downloader_probe.json`: Binance's official downloader commit,
request scope, checksum verification, and timestamp-only comparison result

Could you please confirm whether these candles can be recovered and republish
corrected monthly/daily archives and `.CHECKSUM` files? If the underlying
records are permanently unavailable, an explicit confirmation would also help
downstream users apply a documented missing-data policy.

Thank you.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the attached CSV files, evidence_manifest.json, and official_downloader_probe.json, then inspect the referenced Binance downloader scripts at commit 5c7f3197591c0d54d85dc43066226bc4c671d47a. Verify whether the 603 rows can be recovered from available sources; done means either republishing corrected archives and .CHECKSUM files or documenting that the records are unavailable.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.