alibaba / alibaba/clusterdata

csv file structure corrupted.

Open
#222 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
2.2k
Forks
482
PR merge metrics
No merged PRs in 30d

Description

Hi, I run `bash fetchData.sh start_date=0d0 end_date=1d1` to download the files. And I extract `CallGraph/CallGraph_0.tar.gz` to get `CallGraph_0.csv`.

I use csvq to parse the csv, then I got `parse error around line 58755`, the `rpc_id` of this line is `0.1.1.1,0.1.1.1` without quotes.

```
[root@ha-master-1 CallGraph]# csvq 'select `rpctype`,count(*) from `./CallGraph_0.csv` group by rpctype'
[L:1 C:32] data parse error in /root/code/clusterdata/cluster-trace-microservices-v2022/data/CallGraph/CallGraph_0.csv: line 58755, column 116: wrong number of fields in line
[root@ha-master-1 CallGraph]#
```

![image](https://github.com/user-attachments/assets/3e6d74c2-fb40-4317-814c-2a033ad18757)

And I also find some other strange errors about the data.

![image](https://github.com/user-attachments/assets/e637a197-69ed-4170-9e99-3abf88478198)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running `bash fetchData.sh start_date=0d0 end_date=1d1` and extracting `CallGraph/CallGraph_0.tar.gz`. Inspect `CallGraph_0.csv` around line 58755 and parse it with the reported csvq command, then compare the malformed `rpc_id` and other strange rows with the expected CSV structure. Done means the generated CSV can be parsed without wrong-field-count errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
bash
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.