Cannot import torchdata when installed with conda on WSL

🐛 Describe the bug

$ conda install -c pytorch torchdata -y
$ python -c "import torchdata"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/erip/anaconda3/envs/tmp-dev/lib/python3.10/site-packages/torchdata/__init__.py", line 7, in <module>
    from torchdata import _extension  # noqa: F401
  File "/home/erip/anaconda3/envs/tmp-dev/lib/python3.10/site-packages/torchdata/_extension.py", line 34, in <module>
    _init_extension()
  File "/home/erip/anaconda3/envs/tmp-dev/lib/python3.10/site-packages/torchdata/_extension.py", line 31, in _init_extension
    from torchdata import _torchdata as _torchdata
ImportError: /usr/lib/x86_64-linux-gnu/libp11-kit.so.0: undefined symbol: ffi_type_pointer, version LIBFFI_BASE_7.0
$ conda uninstall torchdata -y && pip install torchdata
$ python -c "import torchdata"
$

Versions

PyTorch version: 1.13.1
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 528.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] torch==1.13.1
[pip3] torchdata==0.5.1
[pip3] torchtext==0.14.1
[conda] blas 1.0 mkl
[conda] mkl 2022.1.0 hc2b9512_224
[conda] pytorch 1.13.1 py3.10_cuda11.7_cudnn8.5.0_0 pytorch
[conda] pytorch-cuda 11.7 h67b0de4_1 pytorch
[conda] pytorch-mutex 1.0 cuda pytorch
[conda] torchdata 0.5.1 pypi_0 pypi
[conda] torchtext 0.14.1 py310 pytorch

Reproduce the conda installation and import command in the reported Ubuntu 20.04 WSL environment, then compare it with the pip installation that succeeds. Start with torchdata/init.py and torchdata/_extension.py, focusing on the reported libp11-kit.so.0 and libffi symbol error; done means the conda-installed package imports successfully.