microsoft / microsoft/FLAML

Support In-Context-Learning (ICL) in agents

Open
#1,064 4 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Jupyter Notebook
Stars
4.4k
Forks
565
Avg merge
5d 8m
Merged PRs (30d)
17

Description

Scenario: Interactive example selection. More specifically, the`AssistantAgent` can ask for examples anytime during the interaction with the `UserProxyAgent`

```[tasklist]
### Tasks
- [x] Review PR https://github.com/microsoft/FLAML/pull/1056
- [x] Write user code example for the above scenario
- [x] Enable exmaple selection (could start with a vanilla example selection method, e.g., random) in `UserProxyAgent`.
```
```[tasklist]
### Tasks: June10-13
- [x] Conduct manual initial experiments to verify whether the LLM can request new examples as planned. The hypothesis is that the LLM can use triggers and prompts to evaluate and request examples on its own. We need to determine whether the LLM possesses this capability.
- [ ] Keep in mind and not urgent: We need to consider whether to add the ICL as a "skill" in the AssistantAgent, aligning it with the UserProxyAgent's skills. One possible solution is to include a prompt in the system message when the UserProxyAgent is detected to be working on ICL.
```

```[tasklist]
### Tasks: June 14-20
- [x] Prompt Engineering: find the best prompt which can elicit the LLM to think of which kind of exemplars are most useful
- [x] Find more datasets @tong
- [x] Implement the first algorithm (LLM information+ coverage), and have some initial results
```

```[tasklist]
### Tasks: June 20-27
- [x] Study the https://arxiv.org/abs/2304.11406 paper, and figure out whether we can use the personalized datasets
- [x] @littlelittlecloud Check the ICL of Math dataset without Voting
- [ ] @tongwu2020 Think about the methods of improving the Math performance
- [x] Manually check the ICL results of the Math dataset
- [x] Ask for the dataset of API bank
```

```[tasklist]
### Tasks: June 27
- [ ] @littlelittlecloud @jtongxin #tongwu2020 Design the experiments for Math
- [ ] @jtongxin Study the performance of GPT-4/3.5 on LaMP dataset
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.