microsoft / microsoft/SPTAG

Can I just search by the metadata?

Open
#96 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

customer raised need_triage question
Dominant language
C++
Stars
5k
Forks
622
Avg merge
8h 37m
Merged PRs (30d)
5

Description

So here's my code. This returns something but I'm not sure why it returns it that way. Can somebody explain? This is based on the sample code from the github site.

`
using Microsoft.ANN.SPTAGManaged;
using System;
using System.IO;
using System.Text;

namespace SPTAG_Tester
{
class Program
{
static int dimension = 2;
static int n = 20;
static int k = 3;
static byte[] createFloatArray(int n)
{
byte[] data;
if (n == 1)
{
data = new byte[n * dimension * sizeof(float)];
var ee = 5;

            for (int i = (0 + ee); i < (n + ee); i++)
            {
                for (int j = 0; j < dimension; j++)
                {
                    var bb = BitConverter.GetBytes((float)i);
                    Array.Copy(bb, 0, data, ((i-ee) * dimension + j) * sizeof(float), 4);
                }
            }

            var l = System.Text.Encoding.ASCII.GetString(data, 0, data.Length);
                            
            return data;
        }

        data = new byte[n * dimension * sizeof(float)];

        for (int i = 0; i < n; i++)
            for (int j = 0; j < dimension; j++)
                Array.Copy(BitConverter.GetBytes((float)i), 0, data, (i * dimension + j) * sizeof(float), 4);

        return data;
    }

    static byte[] createMetadata(int n)
    {
        StringBuilder sb = new StringBuilder();
        sb.Append("zero\t65.0|220.0|\n");
        sb.Append("one\t73.0|160.0|\n");
        sb.Append("two\t59.0|110.0|\n");
        sb.Append("three\t61.0|120.0|\n");
        sb.Append("four\t75.0|150.0|\n");
        sb.Append("five\t67.0|240.0|\n");
        sb.Append("six\t68.0|230.0|\n");
        sb.Append("seven\t70.0|220.0|\n");
        sb.Append("eight\t62.0|130.0|\n");
        sb.Append("nine\t66.0|210.0|\n");
        sb.Append("ten\t77.0|190.0|\n");
        sb.Append("11\t75.0|180.0|\n");
        sb.Append("12\t74.0|170.0|\n");
        sb.Append("13\t70.0|210.0|\n");
        sb.Append("14\t61.0|110.0|\n");
        sb.Append("15\t58.0|100.0|\n");
        sb.Append("16\t66.0|230.0|\n");
        sb.Append("17\t59.0|120.0|\n");
        sb.Append("18\t68.0|210.0|\n");
        sb.Append("19\t61.0|130.0|\n");

        return Encoding.ASCII.GetBytes(sb.ToString());
    }

    static void Main()
    {
        {
            AnnIndex idx = new AnnIndex("BKT", "Float", dimension);
            idx.SetBuildParam("DistCalcMethod", "L2");
            byte[] data = createFloatArray(n);

            byte[] meta = createMetadata(n);
            idx.BuildWithMetaData(data, meta, n, false);
            idx.Save("testcsharp");
            
        }

        AnnIndex index = AnnIndex.Load("testcsharp");
        var toSearch = createFloatArray(1);
                    
        BasicResult[] res = index.SearchWithMetaData(toSearch, k);
        for (int i = 0; i < res.Length; i++)
            Console.WriteLine("result num:" + i.ToString() +
                ": [" + res[i].Dist.ToString() + "(Distance)] " +
                "@( [" + res[i].VID.ToString() + "(VectorID)] , " +
                "[" + Encoding.ASCII.GetString(res[i].Meta).Replace("\n", string.Empty).Replace("\t", " ") + "(Metadata)])");
        Console.WriteLine("test finish!");

        Console.ReadLine();
    }
}

}
`

The result is:
`
result num:0: [0(Distance)] @( [5(VectorID)] , [five 67.0|240.0|(Metadata)])

result num:1: [2(Distance)] @( [4(VectorID)] , [four 75.0|150.0|(Metadata)])

result num:2: [2(Distance)] @( [6(VectorID)] , [six 68.0|230.0|(Metadata)])
`

Why do I need to generate the 5th one in the data to get the VectorID 5? Isn't there a method to just search by the metadata id "five" instead of getting the byte[]'s by offsetting the index by 5?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the managed API entry points AnnIndex.BuildWithMetaData and SearchWithMetaData, then trace how metadata and VectorID are represented in the SPTAG implementation. Determine whether metadata identifiers can be queried directly rather than requiring a vector position, and document the supported behavior or the scope of an API change.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, csharp
Domain
api, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.