facebookresearch / facebookresearch/segment-anything
Sam decoder model is returning the error : Cannot read property 'buffer' of undefined, when i try to load the sam encoder image embeddings with .txt file extension. Using ONNX react-native runtime.
- Dominant language
- Jupyter Notebook
- Stars
- 54.9k
- Forks
- 6.4k
- PR merge metrics
- No merged PRs in 30d
Description
I have the encoder image_embeddings in a text file in my root project directory. When i try to read the text file with encoder embeddings, the react-native can able to read the file. But if i pass the data to the decoder model its not even reading the image_embeddings from the text file. Its returning this error: **Cannot read property 'buffer' of undefined**.
Here is my code please check it.
const decoderModelPath = `${RNFS.TemporaryDirectoryPath}/vit_h_decoder.onnx`;
console.log('Decoder model path is loading....');
await RNFS.downloadFile({
fromUrl: Image.resolveAssetSource(
require('./models/vit_h_decoder.onnx'),
).uri,
toFile: decoderModelPath,
}).promise;
console.log('Decoder model is started processing....');
const decoderSession = await ort.InferenceSession.create(
'file://' + decoderModelPath,
);
console.log('Decoder model is loaded....');
const txtFile = `${RNFS.TemporaryDirectoryPath}/react-embeddings.txt`;
await RNFS.downloadFile({
fromUrl: Image.resolveAssetSource(require('./react-embeddings.txt'))
.uri,
toFile: txtFile,
}).promise;
console.log('Embeddings are loading...Please wait....');
const fileEmbeddings = await FileSystem.readFile(txtFile);
console.log('Embeddings are going to parse....');
const parseDataFile = JSON.parse(fileEmbeddings);
console.log(Object.keys(parseDataFile));
console.log('Embeddings are Parsed successfully....');
console.log('Feed is going to load....');
const feed = {
image_embeddings: parseDataFile,
point_coords: new ort.Tensor(
new Float32Array([10, 10, 0, 0]),
[1, 2, 2],
),
point_labels: new ort.Tensor(new Float32Array([0, -1]), [1, 2]),
mask_input: new ort.Tensor(
new Float32Array(256 * 256),
[1, 1, 256, 256],
),
has_mask_input: new ort.Tensor(new Float32Array([0]), [1]),
orig_im_size: new ort.Tensor(new Float32Array([684, 1024]), [2]),
};
console.log('Feed is loaded...');
const finalData = await decoderSession.run(feed);
const filePath = `${RNFS.DocumentDirectoryPath}/example.txt`;
await RNFS.writeFile(
filePath,
JSON.stringify(finalData.masks.data),
'utf8',
);
await Share.open({
title: 'Share file',
url: `file://${filePath}`,
});
console.log('Done with the decoder model');
Contributor guide
Research direction
Start with the provided React Native flow around vit_h_decoder.onnx, react-embeddings.txt, and decoderSession.run(feed). Check the runtime's documented input tensor requirements and compare them with the parsed image_embeddings value and the other feed tensors. Done means identifying the incompatible input or a reproducible runtime fix, then confirming decoderSession.run completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, react-native
- Domain
- machine-learning, mobile-dev
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100