binance / binance/binance-public-data

ZIP file with nested aggtrades CSV files: which guidelines to follow for consistent reading?

Open
#449 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
613
PR merge metrics
No merged PRs in 30d

Description

Hello,

I am trying to consolidate aggtrades list from aggtrades files.

I have came accross
[ETHUSDT-aggTrades-2018-01.zip](https://data.binance.vision/data/spot/monthly/aggTrades/ETHUSDT/ETHUSDT-aggTrades-2018-01.zip)

This one is made of:
- at root level, a 1.4Go **ETHUSDT-aggTrades-2018-01.csv** (let's call it **A**)
- some nested directories, with ultimately a ~ 341Mo **ETHUSDT-aggTrades-2018-01.csv** (let's call it **B**)

**A** has 4 times the content of **B**, but there is a difference of 1 in **quantity** column.

Please, to get a consistent list of trades (reflecting the "real" trades), what should be done "systematically" in this case?
Should we always try to use the smallest file because we can assume you have cleaned the list of aggtrades?

Why are there 2 files? Is it a leftover from a data cleaning workflow and you will fix this? Or is it a result to be expected from time to time as being part of your process?

For a more robust aggtrade list building, should we read both A and B, concatenate them and then deduplicate?
If we have nearly same rows between A & B, but differing for some values, which one should be favored, A or B?

I am sorry for the numerous questions.
Thank you to help me understanding which workflow should be followed to build a consistent list of aggtrades.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by downloading ETHUSDT-aggTrades-2018-01.zip from the linked Binance archive and compare the root CSV (A) with the nested CSV (B), including their quantity values and row counts. No repository files or tests are named; done means documenting which file is authoritative and the systematic workflow for producing a consistent aggtrades list.

Written by the indexing model from the issue text.

Assessment

Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.