PAIR-code / PAIR-code/deliberate-lab

Add nuanced prompt scaffolding around "profile info" based on profile type

Open
#862 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

runtime:agents stale
Dominant language
TypeScript
Stars
96
Forks
40
Avg merge
2d 11h
Merged PRs (30d)
20

Description

See context and comment in https://github.com/PAIR-code/deliberate-lab/pull/814:

"Update: I ended up adding a set of TODO comments for now as I think this warrants a bigger discussion around profile type specific scaffolding—happy to file an issue to circle back.

If I understand your intention correctly here, your implementation in the original PR does not actually solve this. The profileSetId bit is being used as a hack for hardcoding the profile for the current stage only (I've left a comment in my commit to explain this); we are currently using this for negotiation games where participants interact with different aliases for each game. This means that profileSetId being empty does not indicate whether or not the profile type is assigned/selected; it just means there is no hardcoding for the current stage.

What we'd want to check instead is the ProfileType indicated in the profile stage of the experiment (we have a P2 out to move that field to the experiment level, by the way). Right now, there are two types (default, default gendered) that are "selected" and two types (anonymous animal, anonymous participant) that are "assigned." So the work would involve extracting that bit from the experiment and (likely) storing it in the promptData object that is used throughout functions/src/structured_prompt.utils.ts to fill in the prompt.

Additionally, we may want to think about how we present profile info in the hardcoded cases. If a participant was playing as "Lion" for a past set of stages but will be represented as "Tiger" in the current stage, presumably the scaffolding should communicate this. (We could always share all the profile identities but my guess is that might lead to model confusion / leaking of alternate profile information.)

Anyway, I'll merge this as-is with TODOs and let's figure out what the right approach to "better profile type specific scaffolding" is as a follow-up."

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with functions/src/structured_prompt.utils.ts and trace how experiment profile-stage data reaches the promptData object. Resolve how selected versus assigned ProfileType values should affect scaffolding, including hardcoded profile identities across stages. Done means the agreed profile-specific behavior is implemented without leaking alternate identities, with tests covering the resulting prompt content.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.