[QST] Install Transformers4Rec on Databricks

❓ Questions & Help

Details

I was able to follow the tutorial for installing NVTabular on Databricks, and I was wondering if the same set of instructions or similar exists for Transformers4Rec. Seems like there are a couple of approaches for installing Transformers4Rec on Databricks:

1. Build the NVTabular Databricks Docker image and conda install -c nvidia transformers4rec within a notebook
2. Modify the Dockerfile to download Transformers4Rec, in the same way that NVTabular is in README by echo "https://conda.anaconda.org/nvidia/linux-64/nvtabular-0.6.1-py38_0.tar.bz2" >> docker/rapids-spec.txt

I tried approach #1 but had incompatible library issues, and I'm currently trying out option #2. However, I'm concerned with incompatibility issues between Transformers4Rec, NVTabular, and even the Databricks runtime -- so I'm wondering if there's an "ideal" way to use a Transformers4Rec Docker image on Databricks.

Start with the linked NVTabular Databricks guide, the Docker instructions in README, and docker/rapids-spec.txt. Compare the proposed installation approaches and identify the compatibility constraints involving Transformers4Rec, NVTabular, and the Databricks runtime. Done means documenting a supported installation path and its required versions.