dandi / dandi/dandi-medit

Keywords imported from external sources may be too broad or irrelevant

Open
#81 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
3
Forks
3
Avg merge
6h 37m
Merged PRs (30d)
46

Description

## Problem

When importing metadata from a preprint, the keywords applied to the dandiset may not be specific or relevant to the dataset. During testing with dandiset 001278, keywords like "mental disorder" and "movement disorder" were added but were not well-suited to the dataset's actual focus. It was unclear whether these came from OpenAlex's keyword data or were generated by the LLM.

Additionally, the LLM did not import all available keywords — it selectively included some while omitting others.

## Current Behavior

The system prompt already instructs: "Are keywords specific and informative? (NOT broad generic terms like 'neuroscience', 'brain', 'data')". However, this guidance is not always followed.

The OpenAlex `select` parameter in `fetchUrl.ts` includes `keywords`, so these are available in the fetched data. But the LLM may either import irrelevant ones from OpenAlex or fabricate its own.

## Proposed Solution

Options (not mutually exclusive):
1. **Better prompt guidance**: Instruct the LLM to critically evaluate keywords and only include those directly relevant to the dataset, not just the general paper topic. Also instruct it to present candidate keywords to the user for approval.
2. **Prefer paper-defined keywords**: If the paper itself lists keywords (e.g., in the abstract or metadata), prefer those over computed/assigned keywords.
3. **Deterministic extraction**: If keywords come from OpenAlex, extract them programmatically (similar to the proposed author script) and present all of them to the user for selection.

## Context

Observed during testing with dandiset 001278.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in fetchUrl.ts, where the OpenAlex select parameter includes keywords, and read the current system prompt guidance on keyword specificity. Reproduce the dandiset 001278 behavior and compare imported OpenAlex keywords with those selected or generated by the LLM. Done requires an agreed approach for filtering or presenting candidate keywords, with the chosen behavior verified against this case.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.