allenai / allenai/allenact

Cannot reproduce the performance of EmbCLIP on the ObjectNav task of the RoboTHOR

未关闭
#377 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
383
派生
59
PR 合并指标
30 天内没有已合并 PR

描述

## Problem / Question

Official EmbCLIP is on the leaderboard with a TestSPL score of 0.2.
But according to the [tutorial](https://github.com/allenai/embodied-clip/blob/main/readme_files/baselines_robothor_objectnav.md), I got a score of 0.17.
The command I ran was:
```
python allenact/main.py -o storage/objectnav-robothor-rgb-clip-rn50 -b projects/objectnav_baselines/experiments/robothor/clip objectnav_robothor_rgb_clipresnet50gru_ddppo -c pretrained_model_ckpts/objectnav-robothor-clip-rn50.130M.pt
```
And I got the file `storage/objectnav-robothor-rgb-clip-rn50/metrics/ObjectNav-RoboTHOR-RGB-ClipResNet50GRU-DDPPO/2024-02-15_03-32-04/metrics__test_2024-02-15_03-32-04.json`
According to the method in the [documentation](https://allenact.org/projects/objectnav_baselines/), I got the metric of the validation.
According to the scripts form [robothor-challenge](https://github.com/allenai/robothor-challenge/blob/main/robothor_challenge/scripts/convert_allenact_metrics.py), I run the command:
```
python convert_allenact_metrics.py -v storage/objectnav-robothor-rgb-clip-rn50/metrics/ObjectNav-RoboTHOR-RGB-ClipResNet50GRU-DDPPO/2024-02-15_02-36-50/metrics__val_2024-02-15_02-36-50.json -t storage/objectnav-robothor-rgb-clip-rn50/metrics/ObjectNav-RoboTHOR-RGB-ClipResNet50GRU-DDPPO/2024-02-15_03-32-04/metrics__test_2024-02-15_03-32-04.json -o submission_metrics.json.gz
```
Then submit the `submission_metrics.json.gz` file,

![image](https://github.com/allenai/allenact/assets/56984759/f7619f49-2a32-42be-a2db-cf1fa21243f5)

What should I do?

## Additional context

_(Optional)_ - To provide support it's helpful to have as many details as possible, add additional context here.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。