facebookresearch / facebookresearch/segment-anything

Sam decoder model is returning the error : Cannot read property 'buffer' of undefined, when i try to load the sam encoder image embeddings with .txt file extension. Using ONNX react-native runtime.

Open
#726 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
54.9k
Forks
6.4k
PR merge metrics
No merged PRs in 30d

Description

I have the encoder image_embeddings in a text file in my root project directory. When i try to read the text file with encoder embeddings, the react-native can able to read the file. But if i pass the data to the decoder model its not even reading the image_embeddings from the text file. Its returning this error: **Cannot read property 'buffer' of undefined**.

Here is my code please check it.

const decoderModelPath = `${RNFS.TemporaryDirectoryPath}/vit_h_decoder.onnx`;

console.log('Decoder model path is loading....');

await RNFS.downloadFile({
fromUrl: Image.resolveAssetSource(
require('./models/vit_h_decoder.onnx'),
).uri,
toFile: decoderModelPath,
}).promise;

console.log('Decoder model is started processing....');

const decoderSession = await ort.InferenceSession.create(
'file://' + decoderModelPath,
);

console.log('Decoder model is loaded....');

const txtFile = `${RNFS.TemporaryDirectoryPath}/react-embeddings.txt`;

await RNFS.downloadFile({
fromUrl: Image.resolveAssetSource(require('./react-embeddings.txt'))
.uri,
toFile: txtFile,
}).promise;

console.log('Embeddings are loading...Please wait....');

const fileEmbeddings = await FileSystem.readFile(txtFile);

console.log('Embeddings are going to parse....');

const parseDataFile = JSON.parse(fileEmbeddings);

console.log(Object.keys(parseDataFile));
console.log('Embeddings are Parsed successfully....');

console.log('Feed is going to load....');

const feed = {
image_embeddings: parseDataFile,
point_coords: new ort.Tensor(
new Float32Array([10, 10, 0, 0]),
[1, 2, 2],
),
point_labels: new ort.Tensor(new Float32Array([0, -1]), [1, 2]),
mask_input: new ort.Tensor(
new Float32Array(256 * 256),
[1, 1, 256, 256],
),
has_mask_input: new ort.Tensor(new Float32Array([0]), [1]),
orig_im_size: new ort.Tensor(new Float32Array([684, 1024]), [2]),
};

console.log('Feed is loaded...');
const finalData = await decoderSession.run(feed);

const filePath = `${RNFS.DocumentDirectoryPath}/example.txt`;

await RNFS.writeFile(
filePath,
JSON.stringify(finalData.masks.data),
'utf8',
);

await Share.open({
title: 'Share file',
url: `file://${filePath}`,
});

console.log('Done with the decoder model');

Contributor guide

Open the contributing guide

Research direction

Start with the provided React Native flow around vit_h_decoder.onnx, react-embeddings.txt, and decoderSession.run(feed). Check the runtime's documented input tensor requirements and compare them with the parsed image_embeddings value and the other feed tensors. Done means identifying the incompatible input or a reproducible runtime fix, then confirming decoderSession.run completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, react-native
Domain
machine-learning, mobile-dev
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.