lmcinnes / lmcinnes/umap

TypeError: Descriptors cannot not be created directly. - protocol buffer bug

Open
#1,003 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.3k
Forks
871
Avg merge
1d 13h
Merged PRs (30d)
5

Description

Today I got an expected error, when trying to:
import umap

Error trace is below.

I tried to install the version of PROTOCOL_BUFFERS

!pip install protobuf==3.19
!pip install protobuf==3.20
!pip install protobuf

with no success.

It seems it conflicts with Tensorflow.
Tensorflow version installed is 2.11

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[18], line 16
     13 import librosa
     15 # Clustering
---> 16 import umap
     17 import umap.plot
     19 import hdbscan

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/umap/__init__.py:7
      5     with catch_warnings():
      6         simplefilter("ignore")
----> 7         from .parametric_umap import ParametricUMAP
      8 except ImportError:
      9     warn(
     10         "Tensorflow not installed; ParametricUMAP will be unavailable",
     11         category=ImportWarning,
     12     )

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/umap/parametric_umap.py:14
     10 import sys
     13 try:
---> 14     import tensorflow as tf
     15 except ImportError:
     16     warn(
     17         """The umap.parametric_umap package requires Tensorflow > 2.0 to be installed.
     18     You can install Tensorflow at https://www.tensorflow.org/install
   (...)
     24     """
     25     )

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/__init__.py:37
     34 import sys as _sys
     35 import typing as _typing
---> 37 from tensorflow.python.tools import module_util as _module_util
     38 from tensorflow.python.util.lazy_loader import LazyLoader as _LazyLoader
     40 # Make sure code inside the TensorFlow codebase can use tf2.enabled() at import.

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/python/__init__.py:37
     29 # We aim to keep this file minimal and ideally remove completely.
     30 # If you are adding a new file with @tf_export decorators,
     31 # import it in modules_with_exports.py instead.
     32 
     33 # go/tf-wildcard-import
     34 # pylint: disable=wildcard-import,g-bad-import-order,g-import-not-at-top
     36 from tensorflow.python import pywrap_tensorflow as _pywrap_tensorflow
---> 37 from tensorflow.python.eager import context
     39 # pylint: enable=wildcard-import
     40 
     41 # Bring in subpackages.
     42 from tensorflow.python import data

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/python/eager/context.py:28
     25 from absl import logging
     26 import numpy as np
---> 28 from tensorflow.core.framework import function_pb2
     29 from tensorflow.core.protobuf import config_pb2
     30 from tensorflow.core.protobuf import coordination_config_pb2

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/core/framework/function_pb2.py:16
     11 # @@protoc_insertion_point(imports)
     13 _sym_db = _symbol_database.Default()
---> 16 from tensorflow.core.framework import attr_value_pb2 as tensorflow_dot_core_dot_framework_dot_attr__value__pb2
     17 from tensorflow.core.framework import node_def_pb2 as tensorflow_dot_core_dot_framework_dot_node__def__pb2
     18 from tensorflow.core.framework import op_def_pb2 as tensorflow_dot_core_dot_framework_dot_op__def__pb2

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/core/framework/attr_value_pb2.py:16
     11 # @@protoc_insertion_point(imports)
     13 _sym_db = _symbol_database.Default()
---> 16 from tensorflow.core.framework import tensor_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__pb2
     17 from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
     18 from tensorflow.core.framework import types_pb2 as tensorflow_dot_core_dot_framework_dot_types__pb2

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_pb2.py:16
     11 # @@protoc_insertion_point(imports)
     13 _sym_db = _symbol_database.Default()
---> 16 from tensorflow.core.framework import resource_handle_pb2 as tensorflow_dot_core_dot_framework_dot_resource__handle__pb2
     17 from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
     18 from tensorflow.core.framework import types_pb2 as tensorflow_dot_core_dot_framework_dot_types__pb2

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/core/framework/resource_handle_pb2.py:16
     11 # @@protoc_insertion_point(imports)
     13 _sym_db = _symbol_database.Default()
---> 16 from tensorflow.core.framework import tensor_shape_pb2 as tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2
     17 from tensorflow.core.framework import types_pb2 as tensorflow_dot_core_dot_framework_dot_types__pb2
     20 DESCRIPTOR = _descriptor.FileDescriptor(
     21   name='tensorflow/core/framework/resource_handle.proto',
     22   package='tensorflow',
   (...)
     26   ,
     27   dependencies=[tensorflow_dot_core_dot_framework_dot_tensor__shape__pb2.DESCRIPTOR,tensorflow_dot_core_dot_framework_dot_types__pb2.DESCRIPTOR,])

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/tensorflow/core/framework/tensor_shape_pb2.py:36
     13 _sym_db = _symbol_database.Default()
     18 DESCRIPTOR = _descriptor.FileDescriptor(
     19   name='tensorflow/core/framework/tensor_shape.proto',
     20   package='tensorflow',
   (...)
     23   serialized_pb=_b('\n,tensorflow/core/framework/tensor_shape.proto\x12\ntensorflow\"z\n\x10TensorShapeProto\x12-\n\x03\x64im\x18\x02 \x03(\x0b\x32 .tensorflow.TensorShapeProto.Dim\x12\x14\n\x0cunknown_rank\x18\x03 \x01(\x08\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\tB\x87\x01\n\x18org.tensorflow.frameworkB\x11TensorShapeProtosP\x01ZSgithub.com/tensorflow/tensorflow/tensorflow/go/core/framework/tensor_shape_go_proto\xf8\x01\x01\x62\x06proto3')
     24 )
     29 _TENSORSHAPEPROTO_DIM = _descriptor.Descriptor(
     30   name='Dim',
     31   full_name='tensorflow.TensorShapeProto.Dim',
     32   filename=None,
     33   file=DESCRIPTOR,
     34   containing_type=None,
     35   fields=[
---> 36     _descriptor.FieldDescriptor(
     37       name='size', full_name='tensorflow.TensorShapeProto.Dim.size', index=0,
     38       number=1, type=3, cpp_type=2, label=1,
     39       has_default_value=False, default_value=0,
     40       message_type=None, enum_type=None, containing_type=None,
     41       is_extension=False, extension_scope=None,
     42       serialized_options=None, file=DESCRIPTOR),
     43     _descriptor.FieldDescriptor(
     44       name='name', full_name='tensorflow.TensorShapeProto.Dim.name', index=1,
     45       number=2, type=9, cpp_type=9, label=1,
     46       has_default_value=False, default_value=_b("").decode('utf-8'),
     47       message_type=None, enum_type=None, containing_type=None,
     48       is_extension=False, extension_scope=None,
     49       serialized_options=None, file=DESCRIPTOR),
     50   ],
     51   extensions=[
     52   ],
     53   nested_types=[],
     54   enum_types=[
     55   ],
     56   serialized_options=None,
     57   is_extendable=False,
     58   syntax='proto3',
     59   extension_ranges=[],
     60   oneofs=[
     61   ],
     62   serialized_start=149,
     63   serialized_end=182,
     64 )
     66 _TENSORSHAPEPROTO = _descriptor.Descriptor(
     67   name='TensorShapeProto',
     68   full_name='tensorflow.TensorShapeProto',
   (...)
    100   serialized_end=182,
    101 )
    103 _TENSORSHAPEPROTO_DIM.containing_type = _TENSORSHAPEPROTO

File /data0/home/h21/luas6629/venv/lib/python3.10/site-packages/google/protobuf/descriptor.py:560, in FieldDescriptor.__new__(cls, name, full_name, index, number, type, cpp_type, label, default_value, message_type, enum_type, containing_type, is_extension, extension_scope, options, serialized_options, has_default_value, containing_oneof, json_name, file, create_key)
    554 def __new__(cls, name, full_name, index, number, type, cpp_type, label,
    555             default_value, message_type, enum_type, containing_type,
    556             is_extension, extension_scope, options=None,
    557             serialized_options=None,
    558             has_default_value=True, containing_oneof=None, json_name=None,
    559             file=None, create_key=None):  # pylint: disable=redefined-builtin
--> 560   _message.Message._CheckCalledFromGeneratedFile()
    561   if is_extension:
    562     return _message.default_pool.FindExtensionByName(full_name)

TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with import umap and inspect umap/__init__.py and umap/parametric_umap.py, where the TensorFlow import begins. Check the project's dependency metadata and supported TensorFlow/protobuf combinations; done means the reported environment can import UMAP without the descriptor TypeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.