tensorflow-text not compatible with Conda -- NotFound error upon import

I am using tensorflow-gpu (2.4.1) with Python 3.7 and CUDA 10.1 on Ubuntu 20.04 with miniconda3. I install tensorflow-text with
pip install tensorflow-text==2.4.1 and it completes without error. When I try to import the module it throws the below error,
However, if I do not specify tensorflow-text version it installs tensorflow 2.5 (CPU) version and works on it. (removing the 2.4.1 GPU version)

!pip install tensorflow-text==2.4.1
!pip install tf-models-official==2.4

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  
import numpy as np
from official.nlp import optimization  

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
<ipython-input-5-9ea998addbe5> in <module>
      1 import tensorflow as tf
      2 import tensorflow_hub as hub
----> 3 import tensorflow_text as text  # Needed for loading universal-sentence-encoder-cmlm/multilingual-preprocess
      4 import numpy as np
      5 from official.nlp import optimization  # to create AdamW optimizer

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/__init__.py in <module>
     19 # pylint: disable=wildcard-import
     20 from tensorflow_text.python import keras
---> 21 from tensorflow_text.python import metrics
     22 from tensorflow_text.python.ops import *
     23 

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/__init__.py in <module>
     18 
     19 # pylint: disable=wildcard-import
---> 20 from tensorflow_text.python.metrics.text_similarity_metric_ops import *
     21 
     22 # Public symbols in the "tensorflow_text.metrics" package.

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/text_similarity_metric_ops.py in <module>
     26 from tensorflow.python.framework import load_library
     27 from tensorflow.python.platform import resource_loader
---> 28 gen_text_similarity_metric_ops = load_library.load_op_library(resource_loader.get_path_to_datafile('_text_similarity_metric_ops.so'))
     29 
     30 

~/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow/python/framework/load_library.py in load_op_library(library_filename)
     55     RuntimeError: when unable to load the library or get the python wrappers.
     56   """
---> 57   lib_handle = py_tf.TF_LoadLibrary(library_filename)
     58   try:
     59     wrappers = _pywrap_python_op_gen.GetPythonWrappers(

NotFoundError: /userdirs/test/miniconda3/envs/tfenvnightly/lib/python3.7/site-packages/tensorflow_text/python/metrics/_text_similarity_metric_ops.so: undefined symbol: _ZNK10tensorflow8OpKernel11TraceStringERKNS_15OpKernelContextEb

Reproduce the import failure in the reported Python 3.7, TensorFlow 2.4.1 GPU, CUDA 10.1 and Ubuntu 20.04 environment. Start with tensorflow_text/python/metrics/text_similarity_metric_ops.py and the reported _text_similarity_metric_ops.so loading path; done means tensorflow_text imports without the undefined-symbol error.