tensorflow / tensorflow/tensorboard

Convertion of vectors to .bytes for standalone embedding projector

Open
#4,166 0 comments 0 reactions 1 assignee View on GitHub

Nobody has claimed this yet.

plugin:projector stat:awaiting tensorflower type:support
Dominant language
TypeScript
Stars
7.2k
Forks
1.7k
Avg merge
4d 22h
Merged PRs (30d)
1

Description

Trying to use my own vectors/labels into the standalone embedding projector instead of the example model.
I did so as a comment below suggests and with following two files:

  1. vectors.bytes which is created with numpy's tofile() command:
array([[ 2.5136387 , -2.084486  ,  0.23319365],
       [ 4.008879  ,  1.2343712 , -1.1920882 ],
       [ 3.4046457 , -0.42918122, -0.84252584],
       ...,
       [ 3.2287045 , -0.49425614, -0.7298379 ],
       [ 0.9056233 , -0.4525969 ,  2.068056  ],
       [ 2.942347  ,  0.94748515, -0.8100381 ]], dtype=float32)

vectors.tofile(os.path.join(LOG_DIR, "tensors.bytes")) (array size is 5000x3)

  1. meta.tsv which is a one column tab seperated file containing 5000 labels.

in standalone_projector_config.json I added

{
  "embeddings": [
    {
      "tensorName": "Doc2Vec Lipid",
      "tensorShape": [
        5000,
        3
      ],
      "tensorPath": "/PATH/tensors.bytes",
      "metadataPath": "/PATH/meta.tsv"
    },
...
}

When uploading the vectors/meta files, in .TSV file format to embedding projector (either online or standalone), everything works as intended but including them in bytes/TSV format I get the following error message:

Bazel Closure Rules</h1><h3>//tensorboard/plugins/projector/vz_projector:standalone</h3><p>No srcs found in transitive closure with path component prefix matching request path.

@harveyslash The .bytes files are binary files filled with a bunch of 32 bit floating point numbers. You can use a numpy array with dtype=numpy.float32 and tofile to generate them:

vectors = numpy.zeros(vector_shape, dtype=numpy.float32)
vectors.tofile('my_tensors.bytes')

Here's a complete example of exporting spaCy Vocab vectors for use in the standalone projector.
https://github.com/explosion/spaCy/blob/master/examples/vectors_tensorboard_standalone.py

Originally posted by @justindujardin in https://github.com/tensorflow/tensorflow/issues/7562#issuecomment-376606027

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.