MaartenGr / MaartenGr/BERTopic
Running BERTopic on Apple Silcon (M1 & M2 chips, ARM64 architecture)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.8k
- Forks
- 920
- Avg merge
- 22h 24m
- Merged PRs (30d)
- 5
Description
## Situation
Apple Silicon chips (M1 & M2) are based on the ARM64 (aka [AArch64](https://apple.stackexchange.com/questions/451238/is-m1-chip-aarch64-or-amd64), not to be confused with AMD64). There are known issues with upstream dependencies for this architecture, for example [numba](https://github.com/numba/numba/issues/5520). You may not always run into this issue, depending on the extras that you need. See also #1014 and #1765.
## Solution: Use VS Code Dev Containers
Using [VS Code Dev Containers](https://code.visualstudio.com/docs/devcontainers/containers) allows you to setup a Linux-based environment. To run BERTopic effectively you need to be aware of two things:
- Make sure to use a Docker image specifically compiled for ARM64
- Make sure to use `volume` instead of `mount-bind`, since the latter significantly reduces I/O speeds to disk
Using https://github.com/b-data/data-science-devcontainers address this issue out of the box (kudos to @benz0li). The propose workflow is as follows:
- Install `python-base` or `python-scipy` [devcontainer](https://github.com/b-data/data-science-devcontainers)
- Open VS Code, build the container
- Work in `/home/vscode` or `/workspaces` as these locations are persisted. For example, using `pip install --user bertopic` installs all Python packages persistently.
## Issues
Using Python 3.12 with the `data-science-devcontainer` to run the following
```python
from sklearn.datasets import fetch_20newsgroups
from sentence_transformers import SentenceTransformer
from bertopic import BERTopic
from umap import UMAP
# Prepare embeddings
docs = fetch_20newsgroups(subset="all", remove=("headers", "footers", "quotes"))["data"]
sentence_model = SentenceTransformer("all-MiniLM-L6-v2")
embeddings = sentence_model.encode(docs, show_progress_bar=True)
# Train BERTopic
topic_model = BERTopic(verbose=True).fit(docs, embeddings)
# Reduce dimensionality of embeddings, this step is optional
reduced_embeddings = UMAP(
n_neighbors=10, n_components=2, min_dist=0.0, metric="cosine"
).fit_transform(embeddings)
# Run the visualization with the original embeddings
# topic_model.visualize_document_datamap(docs, embeddings=embeddings)
# Or, if you have reduced the original embeddings already:
fig = topic_model.visualize_document_datamap(
docs, reduced_embeddings=reduced_embeddings
)
```
Yields the following error
```
2024-02-27 09:08:11,438 - BERTopic - Dimensionality - Fitting the dimensionality reduction algorithm
2024-02-27 09:08:30,791 - BERTopic - Dimensionality - Completed ✓
2024-02-27 09:08:30,792 - BERTopic - Cluster - Start clustering the reduced embeddings
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
2024-02-27 09:08:31,927 - BERTopic - Cluster - Completed ✓
2024-02-27 09:08:31,933 - BERTopic - Representation - Extracting topics from clusters using representation models.
2024-02-27 09:08:34,449 - BERTopic - Representation - Completed ✓
__The Kernel crashed while executing code in the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure.__
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Python 3.12 example from the issue in the referenced ARM64 data-science-devcontainer, using an ARM64 image and a volume under /home/vscode or /workspaces. Review the crash point after clustering and compare the related reports #1014 and #1765. Done means identifying a reproducible dependency or environment cause and documenting a verified setup or next step.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python
- Domain
- devops, infrastructure, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100