microsoft / microsoft/HydraLab

# hydra里基于LLM的SmartMonkey到底是如何实现有点疑问?

Open
#593 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
1k
Forks
107
Avg merge
1d 10h
Merged PRs (30d)
2

Description

看PR稿里实现了基于LLM的 monkey智能探索,对里面的实现比较感兴趣,看完一头雾水有两个疑问向大神们请教下

##1.center里有几个对openapi的封装,但似乎都没看到实际调用的地方,看代码好像没看到代码里有实质调用LLM模型的地方

##2.agent里SmartMonkey,与LLM模型相关的api,version等参数都在 SmartTestParam这个类里定义,但最终传来传去,实际最后程序运行时是收归到了runSmartTestOnce函数,然后如果能执行到,则是进入 smartTestUtil.runPYFunction(smartTestParam, logger);
看这个函数的执行 最后是在执行一段python脚本,非常不解的是代码实现有必要写这么绕吗?
public String runPYFunction(SmartTestParam smartTestParam, Logger logger) throws Exception {
File smartTestFolder = new File(smartTestParam.getOutputFolder(), Const.SmartTestConfig.RESULT_FOLDER_NAME);
smartTestFolder.mkdir();
String res = null;
String[] runArgs = new String[9];
runArgs[0] = "python";
runArgs[1] = filePath;
runArgs[2] = smartTestParam.apkPath;
runArgs[3] = smartTestParam.deviceInfo;
runArgs[4] = smartTestParam.modelInfo;
runArgs[5] = smartTestParam.testSteps;
runArgs[6] = smartTestParam.stringTextFolder;
runArgs[7] = smartTestFolder.getAbsolutePath();
runArgs[8] = smartTestParam.llmInfo;

    for (String tempArg : runArgs) {
        logger.info(tempArg);
    }
    Process proc = Runtime.getRuntime().exec(runArgs);
    SmartTestLog err = new SmartTestLog(proc.getErrorStream(), logger);
    SmartTestLog out = new SmartTestLog(proc.getInputStream(), logger);
    err.start();
    out.start();
    res = out.getContent();
    proc.waitFor();

    return res;
}

然后这里面的filePath, smartTestParam参数 以及所执行到的python文件 PY_FILE_NAME 都是在 hydralab\common\util 目录的 Const.java里所定义。。感觉有点乱七八糟的代码

##3.上面的 smartTestParam参数,涉及 BERT_MODEL_NAME, TOPIC_MODEL_NAME等模型参数的定义,猜测最终SmartMonkey是基于这两个模型来做决策的么,是在线or离线的Reference?这里能在文档里注明出来么,看的有点云里雾里

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with agent SmartMonkey, SmartTestParam, runSmartTestOnce, and SmartTestUtil.runPYFunction, then trace the filePath and PY_FILE_NAME definitions in hydralab/common/util/Const.java. Document where the OpenAPI and LLM calls occur, how the Python process is invoked, and whether BERT_MODEL_NAME and TOPIC_MODEL_NAME are online or offline dependencies; done means the execution flow and model configuration are clear.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, openapi, python
Domain
ai, testing-qa
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.