[Dist] Workaround for `Invalid scalar type` when `dist.scatter()` boolean tensor

## 🔨Work Item

**IMPORTANT:**
* This template is only for dev team to track project progress. For feature request or bug report, please use the corresponding issue templates.
* DO NOT create a new work item if the purpose is to fix an existing issue or feature request. We will directly use the issue in the project tracker.

Project tracker: https://github.com/orgs/dmlc/projects/2

## Description
Due to a bug in `torch.distributed.scatter()`(see more details here: https://github.com/pytorch/pytorch/issues/90245), we have to convert boolean data into uint8 before scattering and convert it back to boolean afterwards. This is just a workaround and introduced in https://github.com/dmlc/dgl/pull/4930. This workaround should be removed once boolean data scatter is supported natively in torch.

This ticket aims to track this issue.

## Depending work items or issues