Enable Kineto CUPTI on Windows

Some chats from slack

me:

@Gisle Dankel Do you have the context of why static linking is required on windows for libkineto?

Gisle:

It’s not - in fact I don’t think static linking is an option on windows? We prefer to link statically because libcupti is not always installed on the system

me:

I am starting to see the reason behind it after exploring my pytorch installation in C:\Users\guangyunhan\Miniconda3\envs\py37\lib\site-packages\torch. On windows, all CUDA related DLLs are installed int o <...>\site-packages\torch\lib\ , that means those libraries are packaged with whl file, therefore, a 3GB whl for user, but it is convenient.

This explains why

libcupti is not always installed on the system

And by

We prefer to link statically

If I take it right, if we want to be the same convenient as above, we need to provide cupti library to user. Statically linking cupti into main pytorch/libkineto DLL is an option.

Unfortunately, NV only provide us some DLLs under C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\extras\CUPTI\lib64 , so the only viable option is to package cupti library into whl

In summary:
on windows

• also package cupti64*.dll, see https://github.com/pytorch/builder/pull/815
  • Because PyTorch package all CUDA DLLs in whl
• Fix cupti activity timestamp semantic on windows
  • manually convert time since boot --> time since epoch
• Enable CUPTI on Windows, see https://github.com/pytorch/pytorch/pull/62175

Start by reviewing the Windows CUPTI context in this issue and the referenced PyTorch pull request 62175. Check how CUPTI DLL packaging and the existing Windows activity-timestamp fix relate to enabling CUPTI, then verify that Windows profiling can use CUPTI and that the completed checklist items remain valid.