facebookresearch / facebookresearch/ImageBind

Thermal x Vision Support

Open
#37 6 comments 7 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9.1k
Forks
842
PR merge metrics
No merged PRs in 30d

Description

Following [issue 14](https://github.com/facebookresearch/ImageBind/issues/14), I created a small example for thermal embedding. While the Vision x Text and Thermal x Text are working properly, it seems the Vision x Thermal does not yield the correct result.

```
def load_and_transform_thermal_data(thermal_paths, device):
if image_paths is None:
return None

thermal_ouputs = []
for thermal_path in thermal_paths:
data_transform = transforms.Compose(
[
transforms.Resize(
224, interpolation=transforms.InterpolationMode.BICUBIC
),
transforms.CenterCrop(224),
transforms.ToTensor(),
# transforms.Normalize(
# mean=(0.5),
# std=(0.5),
# ),
]
)
with open(thermal_path, "rb") as fopen:
thermal = Image.open(fopen).convert("L")
thermal = data_transform(thermal).to(device)
thermal_ouputs.append(thermal)
return torch.stack(thermal_ouputs, dim=0)
```

And the results are:

```
Vision x Text:
[[9.9997604e-01 2.3943641e-05]
[6.0792509e-06 9.9999392e-01]]
Thermal x Text x :
[[1.0000000e+00 1.2433221e-11]
[2.8220674e-02 9.7177935e-01]]
Vision x Thermal Cosine:
[[0.1554441 0.02945926]
[0.16725276 0.03671783]]
Vision x Thermal Softmax:
[[0.7789999 0.22100005]
[0.7867338 0.21326624]]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.