binance / binance/binance-public-data
Missing USD-M 1m markPriceKlines and indexPriceKlines rows across multiple symbols
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 613
- PR merge metrics
- No merged PRs in 30d
Description
Hello Binance public-data maintainers,
While validating the official USD-M Futures 1-minute mark and index price
archives, I found synchronized missing candle rows across multiple symbols.
## Affected data
- Market: USD-M Futures (`data/futures/um`)
- Data types: `markPriceKlines`, `indexPriceKlines`
- Interval: `1m`
- Symbols: `BTCUSDT`, `ETHUSDT`, `XRPUSDT`, `LINKUSDT`, `SOLUSDT`, `DOGEUSDT`
- Missing rows: **603 symbol/data-type minutes**
- `markPriceKlines`: 305
- `indexPriceKlines`: 298
- Distinct missing open times: **66**
- Contiguous symbol/data-type gap ranges: **48**
| UTC date | Missing symbol/data-type minutes | Distinct missing open times |
|---|---:|---:|
| 2020-01-19 | 232 | 29 |
| 2020-12-17 | 288 | 24 |
| 2022-07-12 | 58 | 10 |
| 2022-07-13 | 1 | 1 |
| 2024-08-12 | 24 | 2 |
## Checks performed
1. Checked 43 affected official monthly ZIP files and verified
their published `.CHECKSUM` files.
2. Checked the corresponding official daily ZIP files:
40 were downloaded with verified checksums and
4 returned `NOT_AVAILABLE`.
3. Queried the exact 48 missing ranges through the public
`/fapi/v1/markPriceKlines` or `/fapi/v1/indexPriceKlines` endpoint. Every
response was the empty JSON array `[]`.
4. Re-ran Binance's own `download-futures-markPriceKlines.py` and
`download-futures-indexPriceKlines.py` scripts at repository commit
`5c7f3197591c0d54d85dc43066226bc4c671d47a` for all affected dates. The scripts downloaded and verified
52 ZIP files, but
recovered **0 of the 603 missing rows**.
5. No interpolation or third-party values were used.
The synchronized gaps include:
- 2020-01-19 13:09–13:37 UTC
- 2020-12-17 07:32–07:55 UTC
- scattered minutes on 2022-07-12 and 2022-07-13
- 2024-08-12 10:02–10:03 UTC
## Attachments
[binance_usdm_mark_index_missing_minutes.csv](https://github.com/user-attachments/files/30496219/binance_usdm_mark_index_missing_minutes.csv)
[binance_usdm_mark_index_gap_ranges.csv](https://github.com/user-attachments/files/30496231/binance_usdm_mark_index_gap_ranges.csv)
[evidence_manifest.json](https://github.com/user-attachments/files/30496234/evidence_manifest.json)
[official_downloader_probe.json](https://github.com/user-attachments/files/30496237/official_downloader_probe.json)
- `binance_usdm_mark_index_missing_minutes.csv`: every missing symbol,
data type, and 1-minute open time
- `binance_usdm_mark_index_gap_ranges.csv`: surrounding present timestamps,
archive URLs/statuses/checksums, exact public API URLs, and empty-response
SHA256 evidence
- `evidence_manifest.json`: counts and SHA256 hashes for the two CSV files
- `official_downloader_probe.json`: Binance's official downloader commit,
request scope, checksum verification, and timestamp-only comparison result
Could you please confirm whether these candles can be recovered and republish
corrected monthly/daily archives and `.CHECKSUM` files? If the underlying
records are permanently unavailable, an explicit confirmation would also help
downstream users apply a documented missing-data policy.
Thank you.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing the attached CSV files, evidence_manifest.json, and official_downloader_probe.json, then inspect the referenced Binance downloader scripts at commit 5c7f3197591c0d54d85dc43066226bc4c671d47a. Verify whether the 603 rows can be recovered from available sources; done means either republishing corrected archives and .CHECKSUM files or documenting that the records are unavailable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100