LGBO的数据处理与 oracle/evaluator 实现
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 35/100
- Issue type
- Documentation
- Clarity
- Needs clarification
- Activity status
- Quiet
- Tech stack
- python
- Domain
- data, machine-learning
Research direction
Start by tracing the Figure 3 GPBO/LGBO experiment path and the HPLC, Cross-barrel, and Concrete data entry points; the issue names no files or tests. Compare the runner, preprocessing, and oracle/evaluator behavior against the reported settings, including coordinate scaling, boundaries, duplicates, and repeated queries. Done means the missing runner/configuration and implementation details are provided or documented well enough to reproduce Figure 3.
Written by the indexing model from the issue text.
Description
您好,感谢您公开 SAIBO 代码。
我正在复现论文:
Unleashing LLMs in Bayesian Optimization: Preference-Guided Framework for Scientific Discovery
目前,公开代码中的 LGBO 核心流程可以正常运行,但我们在复现 HPLC、Cross-barrel 和 Concrete 三个数据集上的实验时,得到的 GPBO 和 LGBO 优化轨迹及最终结果均与论文报告结果存在明显差异。
为了排除 LLM 模型和输出差异的影响,我们进一步使用不调用 LLM 的 GPBO 基线进行排查。目前在仓库中暂未找到 Figure 3 所使用的完整实验 runner、数据预处理代码以及 oracle/evaluator 实现。
我们按照论文描述实现了连续 kNN-IDW oracle:
k=12, p=2, eps=1e-12
以下百分比采用我们的本地定义:
100 × best-so-far / 去重均值 oracle 的最大值
在连续 kNN-IDW evaluator 下,GPBO 的最终平均结果为:
| 数据集 | 连续 kNN-IDW |
|---|---|
| HPLC | 67.23% |
| Cross-barrel | 85.87% |
| Concrete | 77.58% |
该结果与 Figure 3 中的优化轨迹和终点存在较大差异。
作为对照,在 GP、采集函数、预算和随机种子均保持不变的情况下,我们将每轮 acquisition 限制在原始数据表中尚未评估的输入上,结果变为:
| 数据集 | 仅从尚未评估的表内输入中选点 |
|---|---|
| HPLC | 88.45% |
| Cross-barrel | 95.37% |
| Concrete | 96.75% |
该对照表明,复现结果对数据处理方式以及 oracle/evaluator 的具体实现非常敏感,这可能同时影响 GPBO 和 LGBO 的结果。
请问是否方便提供生成 Figure 3 结果时使用的:
- 完整实验 runner 和配置文件;
- HPLC、Cross-barrel 和 Concrete 的数据加载与预处理代码;
- 连续 oracle 的构建与查询代码,包括:
- kNN 距离是在原始坐标还是归一化坐标中计算;
- 搜索边界及边界外查询的处理方式;
- 相同输入对应不同输出时的处理方式;
- exact-match 和重复查询的处理规则;
非常感谢!
- Dominant language
- Python
- Stars
- 12
- Forks
- 3
- PR merge metrics
- No merged PRs in 30d
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Similar issues
-
bug
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
zostera/django-bootstrap4#894 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
use-agent-os/agent-os#3276 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
zephyrproject-rtos/zephyr#119726 ·
-
area/auth bug comp/agent P3 platform/discord type/security
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
NousResearch/hermes-agent#117848 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
zilliztech/memsearch#759 ·