binance / binance/binance-public-data
ZIP file with nested aggtrades CSV files: which guidelines to follow for consistent reading?
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 613
- PR merge metrics
- No merged PRs in 30d
Description
Hello,
I am trying to consolidate aggtrades list from aggtrades files.
I have came accross
[ETHUSDT-aggTrades-2018-01.zip](https://data.binance.vision/data/spot/monthly/aggTrades/ETHUSDT/ETHUSDT-aggTrades-2018-01.zip)
This one is made of:
- at root level, a 1.4Go **ETHUSDT-aggTrades-2018-01.csv** (let's call it **A**)
- some nested directories, with ultimately a ~ 341Mo **ETHUSDT-aggTrades-2018-01.csv** (let's call it **B**)
**A** has 4 times the content of **B**, but there is a difference of 1 in **quantity** column.
Please, to get a consistent list of trades (reflecting the "real" trades), what should be done "systematically" in this case?
Should we always try to use the smallest file because we can assume you have cleaned the list of aggtrades?
Why are there 2 files? Is it a leftover from a data cleaning workflow and you will fix this? Or is it a result to be expected from time to time as being part of your process?
For a more robust aggtrade list building, should we read both A and B, concatenate them and then deduplicate?
If we have nearly same rows between A & B, but differing for some values, which one should be favored, A or B?
I am sorry for the numerous questions.
Thank you to help me understanding which workflow should be followed to build a consistent list of aggtrades.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by downloading ETHUSDT-aggTrades-2018-01.zip from the linked Binance archive and compare the root CSV (A) with the nested CSV (B), including their quantity values and row counts. No repository files or tests are named; done means documenting which file is authoritative and the systematic workflow for producing a consistent aggtrades list.
Written by the indexing model from the issue text.
Assessment
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100