Azure / Azure/AzurePublicDataset

Definition of buckets in AzurePublicDatasetV2

Open
#15 1 comment 5 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
1.2k
Forks
185
PR merge metrics
No merged PRs in 30d

Description

Hi,

Can you please include description of VM core and memory buckets to [AzurePublicDatasetV2](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md) dataset? It is just about including these two URLs in [AzurePublicDatasetLinksV2.txt](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetLinksV2.txt)
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv

I am aware that the exact number of VM cores are not given, as discussed in issue #5, and VMs are put in one of six buckets based on their cores or memory. However, it seems that description of these buckets are "missing", even though they were meant to be released.

I say "missing" (in quotes) because even though description file is not included in [AzurePublicDatasetLinksV2.txt](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetLinksV2.txt) they are available for downloading on Azure Blob Storage. More precisely, [schema.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/schema.csv) mentions that description of the CPU buckets are available at `vm_virtual_core_bucket_definition.csv`, which has two fields: bucket and definition. I blindly constructed a path for this file by appending the file name `vm_virtual_core_bucket_definition.csv` to the [parent path](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2) and I was able to download through the constructed path [vm_virtual_core_bucket_definition.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv).

The `vm_virtual_core_bucket_definition.csv` file has description of six buckets. These descriptions match the bucket labels in "VM Cores Distribution" plot in [jupyter notebook](https://github.com/Azure/AzurePublicDataset/blob/master/analysis/Azure%202019%20Public%20Dataset%20V2%20-%20Trace%20Analysis.ipynb), which is referenced in the main [readme](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md). This matching confirms that the file available through Azure Blob Storage is the correct one.

The same applies to description of memory bucket: `schema.csv` mentions `vm_memory_bucket_definition.csv`, it is not included in `AzurePublicDatasetLinksV2.txt` but is available for download in Azure Blob Storage, here [vm_memory_bucket_definition.csv](https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv).

So, it would be great to update `AzurePublicDatasetLinksV2.txt` file to include URL for both files (to avoid future guesswork by others):
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_virtual_core_bucket_definition.csv
- https://azurecloudpublicdataset2.blob.core.windows.net/azurepublicdatasetv2/vm_memory_bucket_definition.csv

Let me know if you accept pull requests. I'd be happy to include these two URLs in `AzurePublicDatasetLinksV2.txt` by myself and perhaps add a short description of buckets to the main [readme](https://github.com/Azure/AzurePublicDataset/blob/master/AzurePublicDatasetV2.md).

Also, is it accurate to say that
- core range in bucket 6 is `>24` and `<=30`, and
- memory range in bucket 6 is `>64` and `<=70`?

I noticed these lines in [jupyter notebook](https://github.com/Azure/AzurePublicDataset/blob/master/analysis/Azure%202019%20Public%20Dataset%20V2%20-%20Trace%20Analysis.ipynb), that suggest these ranges to be correct:
```
#Transform vmcorecount '>24' bucket to 30 and '>64' to 70
max_value_vmcorecountbucket = 30
max_value_vmmemorybucket = 70
trace_dataframe = trace_dataframe.replace({'vmcorecountbucket':'>24'},max_value_vmcorecountbucket)
trace_dataframe = trace_dataframe.replace({'vmmemorybucket':'>64'},max_value_vmmemorybucket)
```
Or is this transformation just a cosmetic improvement to have the jupyter table datatype as `int`? Having more precise bucket bounds would be helpful.

Finally, is there an external document that describes AzurePublicDatasetV2, like SOSP 2017 [paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/10/Resource-Central-SOSP17.pdf) that describes AzurePublicDatasetV1? It would be useful to reference it in the readme, if any.

Thanks in advance for clarifications!

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.