binance / binance/binance-public-data

Duplicate BTCUSDT Futures Data starting 12 SEP 2022

Open
#203 1 comment 0 reactions 0 assignees View on GitHub
Futures reviewing
Dominant language
Python
Stars
2.5k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

The SEP monthly historic data file contains duplicate AggTrade entries starting with AggTrade event id 1446230265 - a few are listed below (the converted UTC time stamp for the first event in the first pair of duplicates is `2022-09-12 00:00:00.166`; the time stamp of the immediately prior event is `2022-09-11 23:59:59.999`):
`1446230265,21820.30,1.614,2815718001,2815718002,1662940800166,true
1446230265,21820.30,1.614,2815718001,2815718002,1662940800166,true
1446230266,21820.40,0.018,2815718003,2815718003,1662940800200,false
1446230266,21820.40,0.018,2815718003,2815718003,1662940800200,false
1446230267,21820.30,0.221,2815718004,2815718007,1662940800206,true
1446230267,21820.30,0.221,2815718004,2815718007,1662940800206,true
1446230268,21820.40,1.911,2815718008,2815718011,1662940800225,false
1446230268,21820.40,1.911,2815718008,2815718011,1662940800225,false
1446230269,21820.30,0.208,2815718012,2815718013,1662940800282,true
1446230269,21820.30,0.208,2815718012,2815718013,1662940800282,true
1446230270,21820.30,0.005,2815718014,2815718014,1662940802349,true
1446230270,21820.30,0.005,2815718014,2815718014,1662940802349,true `
At this point I have not determined how many consecutive duplicate entries there are, but eventually they do stop.
Please advise when the data will be corrected - this is aggravating, for obvious reasons (leading the list: trust in Binance data integrity).
Note that the checksum computed for the ZIP file after download matches the expected checksum so the file was not corrupted during download.
Thanks.
Jack Copper (Institutional customer)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the SEP monthly historic data ZIP and inspect records around AggTrade event id 1446230265, beginning at the stated UTC timestamp. Determine the full duplicate range and verify the corrected archive's checksum; done means the duplicate entries are removed and the published data is replaced.

Written by the indexing model from the issue text.

Assessment

Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.