Azure / Azure/AzurePublicDataset
Definition of buckets in AzurePublicDatasetV2
- Dominant language
- Jupyter Notebook
- Stars
- 1.2k
- Forks
- 185
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Can you please include description of VM core and memory buckets to [AzurePublicDatasetV2](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md) dataset? It is just about including these two URLs in [AzurePublicDatasetLinksV2.txt](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetLinksV2.txt)
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv
I am aware that the exact number of VM cores are not given, as discussed in issue #5, and VMs are put in one of six buckets based on their cores or memory. However, it seems that description of these buckets are "missing", even though they were meant to be released.
I say "missing" (in quotes) because even though description file is not included in [AzurePublicDatasetLinksV2.txt](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetLinksV2.txt) they are available for downloading on Azure Blob Storage. More precisely, [schema.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/schema.csv) mentions that description of the CPU buckets are available at `vm_virtual_core_bucket_definition.csv`, which has two fields: bucket and definition. I blindly constructed a path for this file by appending the file name `vm_virtual_core_bucket_definition.csv` to the [parent path](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2) and I was able to download through the constructed path [vm_virtual_core_bucket_definition.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv).
The `vm_virtual_core_bucket_definition.csv` file has description of six buckets. These descriptions match the bucket labels in "VM Cores Distribution" plot in [jupyter notebook](https://github.com/Azure/AzurePublicDataset/blob/master/analysis/Azure%202019%20Public%20Dataset%20V2%20-%20Trace%20Analysis.ipynb), which is referenced in the main [readme](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md). This matching confirms that the file available through Azure Blob Storage is the correct one.
The same applies to description of memory bucket: `schema.csv` mentions `vm_memory_bucket_definition.csv`, it is not included in `AzurePublicDatasetLinksV2.txt` but is available for download in Azure Blob Storage, here [vm_memory_bucket_definition.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv).
So, it would be great to update `AzurePublicDatasetLinksV2.txt` file to include URL for both files (to avoid future guesswork by others):
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv
Let me know if you accept pull requests. I'd be happy to include these two URLs in `AzurePublicDatasetLinksV2.txt` by myself and perhaps add a short description of buckets to the main [readme](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md).
Also, is it accurate to say that
- core range in bucket 6 is `>24` and `<=30`, and
- memory range in bucket 6 is `>64` and `<=70`?
I noticed these lines in [jupyter notebook](https://github.com/Azure/AzurePublicDataset/blob/master/analysis/Azure%202019%20Public%20Dataset%20V2%20-%20Trace%20Analysis.ipynb), that suggest these ranges to be correct:
```
#Transform vmcorecount '>24' bucket to 30 and '>64' to 70
max_value_vmcorecountbucket = 30
max_value_vmmemorybucket = 70
trace_dataframe = trace_dataframe.replace({'vmcorecountbucket':'>24'},max_value_vmcorecountbucket)
trace_dataframe = trace_dataframe.replace({'vmmemorybucket':'>64'},max_value_vmmemorybucket)
```
Or is this transformation just a cosmetic improvement to have the jupyter table datatype as `int`? Having more precise bucket bounds would be helpful.
Finally, is there an external document that describes AzurePublicDatasetV2, like SOSP 2017 [paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/Resource-Central-SOSP17.pdf) that describes AzurePublicDatasetV1? It would be useful to reference it in the readme, if any.
Thanks in advance for clarifications!
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.