Wrong file path in site-packages

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Current Behavior

```sh
$ git clone --depth=1 --recurse-submodules https://github.com/abetlen/llama-cpp-python
$ cd llama-cpp-python
$ pyproject-build
$ python -m installer --prefix=/tmp/yyy llama_cpp_python-0.2.11-cp311-cp311-manylinux_2_38_x86_64.whl
$ ls /tmp/yyy/lib/python3.11/site-packages
bin include lib llama_cpp llama_cpp_python-0.2.11.dist-info
```

# Expected Behavior

`bin`, `include`, `lib` shouldn't occur in `site-packages`!