facebookresearch / facebookresearch/ImageBind
Thermal x Vision Support
- Dominant language
- Python
- Stars
- 9.1k
- Forks
- 842
- PR merge metrics
- No merged PRs in 30d
Description
Following [issue 14](https://github.com/facebookresearch/ImageBind/issues/14), I created a small example for thermal embedding. While the Vision x Text and Thermal x Text are working properly, it seems the Vision x Thermal does not yield the correct result.
```
def load_and_transform_thermal_data(thermal_paths, device):
if image_paths is None:
return None
thermal_ouputs = []
for thermal_path in thermal_paths:
data_transform = transforms.Compose(
[
transforms.Resize(
224, interpolation=transforms.InterpolationMode.BICUBIC
),
transforms.CenterCrop(224),
transforms.ToTensor(),
# transforms.Normalize(
# mean=(0.5),
# std=(0.5),
# ),
]
)
with open(thermal_path, "rb") as fopen:
thermal = Image.open(fopen).convert("L")
thermal = data_transform(thermal).to(device)
thermal_ouputs.append(thermal)
return torch.stack(thermal_ouputs, dim=0)
```
And the results are:
```
Vision x Text:
[[9.9997604e-01 2.3943641e-05]
[6.0792509e-06 9.9999392e-01]]
Thermal x Text x :
[[1.0000000e+00 1.2433221e-11]
[2.8220674e-02 9.7177935e-01]]
Vision x Thermal Cosine:
[[0.1554441 0.02945926]
[0.16725276 0.03671783]]
Vision x Thermal Softmax:
[[0.7789999 0.22100005]
[0.7867338 0.21326624]]
```
Contributor guide
Assessment
This issue has not been assessed yet.